Semantically-Enriched Pervasive Sensor-Driven Systems1

Semantic Web 1 (2012) 1–5 1 IOS Press

Semantically-enriched Pervasive Sensor-driven Systems 1

Editor(s): Name Surname, University, Country Solicited review(s): Name Surname, University, Country Open review(s): Name Surname, University, Country

Juan Ye a,∗, Stamatia Dasiopoulou b,∗∗, Graeme Stevenson a, Georgios Meditskos b, Vasiliki Efstathiou b, Ioannis Kompatsiaris b, Simon Dobson a, a School of Computer Science, University of St Andrews, UK E-mail:{juan.ye, graeme.stevenson, simon.dobson}.st-andrews.ac.uk b Information Technologies Institute, Centre of Research and Technology Hellas, Greece E-mail:{dasiop, gmeditsk, vefstathiou, ikom}@iti.gr

Abstract. Pervasive and sensor-driven systems are by their nature open and extensible, both in terms of their inputs and the tasks they are required to perform. The data streams coming from sensors are inherently noisy, imprecise, and inaccurate, with differing sampling rates and complex correlations with each other: characteristics that challenge traditional approaches to storing, representing, exchanging, manipulating and programming with rich sensor data. Semantic Web technologies allow designers to capture these properties within a uniform framework. The powerful reasoning techniques with such a representation facility have proven to be attractive in addressing issues such as context modelling and reasoning, service discovery, privacy and trust. In this paper we review the application of the Semantic Web to pervasive and sensor-driven systems. We analyse the strengths and weaknesses of current and projected approaches, and derive a roadmap for using the Semantic Web as a platform on which open, standards-based pervasive, adaptive, and sensor-driven systems can be constructed.

Keywords: Semantic Web, Ontologies, Pervasive Computing, Sensor-Driven Systems, Context Modelling, Context Reasoning, Situation Recognition, Uncertainty, Provenance, Event Modelling, Semantic Service Discovery, Privacy and Trust, Programming

1. Introduction driven and adaptive. Pervasive computing assumes an environment saturated with new classes of intelligent The vision of pervasive computing was ﬁrst artic- and portable devices with sensing, computation and ulated by Mark Weiser in 1991: ‘the most profound communication capabilities. With the help of these technologies are those that disappear. They weave devices, a pervasive system can gather information themselves into the fabric of everyday life until they about users and their surrounding environments and – are indistinguishable from it’ [212]. Two essential without explicit instructions from users – provide cus- characteristics of this vision are that it is both sensor- tomised services that adapt to the users’ current contexts and needs. To be effective, pervasive computing 1This work has been supported by the EU FP7 projects SAPERE: Self-aware Pervasive Service Ecosystems under contract No. 256873 must be supported by an open and standard-based rep- and Dem@Care: Dementia Ambient Care – Multi-Sensing Monitor- resentation so as to facilitate integrating information of ing for Intelligent Remote Management and Decision Support under heterogeneous types and modalities, as well as com- contract No. 288199. *Corresponding author. municating and exchanging information between de- **Corresponding author. vices and components.

The key features of the Semantic Web, namely its sor data, context (information about when, where, who, ability to formally capture the intended semantics and and what, events), reasoning towards higher-level con- to support automated reasoning, allow to elegantly text abstractions, uncertainty handling, semantic ser- share, integrate and manage knowledge coming from vice discovery, and privacy and trust. Section 6 identi- heterogeneous sources, a common requirement in per- fies open research issues in pervasive computing which vasive, sensor-driven, and adaptive computing applica- may potentially be addressed through the use of Se- tions. Rendering meaning explicitly, the Semantic Web mantic Web technologies. Section 7 concludes the pa- facilitates the exchange of data between systems and per. components in an open, extensible manner, maintaining semantic clarity across applications. Semantic Web technologies have successfully addressed some perva- 2. Background of Pervasive Computing sive computing application concerns, such as representing complex sensor data [135], recognising human Pervasive computing aims to enable and assist peo- activities [41], and modelling and querying location ple in accomplishing an increasing number of per- data across heterogeneous coordinate systems [179]. sonal and professional transactions using intelligent Ontological reasoning has proven to be useful in ma- and portable appliances, devices, and artefacts that al- nipulating structured conceptual spaces [185,215,15]. low them to employ intelligent networks and gain di- The use of ontologies, especially of upper and lower rect, straightforward, and secure access to both rele- ontologies with appropriate connections [177], power- vant information and services. It gives people access to fully supports co-operation of data sources within an information stored across potentially large-scale static open system. and ad-hoc networks, allowing them to easily take ac- However, the potential of Semantic Web technolo- tions anywhere and at any time. In the following, we gies to address other key pervasive computing applica- introduce representative application areas and research tion requirements is yet to be fully explored. Open is- themes in pervasive computing and elicit requirements sues include capturing the temporal semantics of data; of building a pervasive system. We also describe the reasoning in the presence of extensive uncertainty; typical information needs of a pervasive system and querying and applying different reasoning schemes to discuss the requirements and challenges to modelling highly dynamic data; capturing provenance – and in- and reasoning on it. tegrating all these elements within a single, coherent framework. In this paper we explore these open issues 2.1. Example Applications and seek to answer three questions for each: to what extent do existing Semantic Web technologies address Pervasive computing has matured from its origins the requirements?, what additional techniques might as an academic research topic to a commercial reality. be needed?, and how might the research community Its potential applications range from intelligent offices address these deficiencies?. to smart homes, healthcare, transportation, and envi- We structure our discussion as follows. Section 2 ronment monitoring. Here we outline example applica- provides a background to pervasive sensing systems, tions to provide a general pervasive computing back- including representative examples of applications and ground. the underlying research themes. It further describes and analyses the characteristics and relationships of 2.1.1. Healthcare and Smart Homes ‘typical’ information that needs to be modelled, in- Pervasive healthcare is an emerging research dis- cluding sensor data, context, events, domain knowl- cipline, focusing on the development and application edge and situations. Section 3 briefly introduces the of pervasive computing technologies for healthcare Semantic Web and highlights the key enabling features and wellbeing. This research is driven by the pressing for knowledge capture and reasoning in the open envi- needs of an expanding ageing population, a predicted ronment of pervasive systems. Section 4 describes rep- shortage in caregivers for the elderly, and the matura- resentative ontology-based models for pervasive com- tion of sensing and communication technologies [142]. puting, showing how and where Semantic Web tech- Pervasive computing technologies introduce new di- nologies are employed. Section 5 analyses the use of agnostic and monitoring methods that directly con- Semantic Web technologies in terms of key pervasive tribute to improvements in care, therapy and medical application requirements, including modelling of sen- treatment. These examples involve sensors and mon- Ye et alia / Semantically-enriched pervasive sensor-driven systems 3 itoring devices that collect and disseminate informa- GPS-enabled devices like smartphones are chang- tion, such as heart rate, blood pressure and glucose lev- ing the ways in which people interact with the Web by els, to healthcare providers [21]. They support a bet- using their locations as context. Multiple users record ter understanding of facets of patients’ daily lives, sup- their outdoor movements as GPS trajectories for travel porting appropriate adaptation of therapies to the in- experience sharing, life logging, sports activity analy- dividual. One scenario is a hospital where a patient is sis, and multimedia content management. This results constantly monitored, and findings linked to a diag- in a large accumulation of GPS trajectories that are nostic process. Thus, it is possible to advise the hos- accumulating unobtrusively and continuously in Web pital canteen to prepare special food for this particu- communities [225]. One use of these data is to discover lar patient and to adapt the patient’s specific medica- interesting locations or location sequences as tourist tion according to his current health condition. Perva- recommendations. Here ‘interesting’ means that a lo- sive computing technologies can also improve medication is not only visited frequently by tourists but also cal treatment procedures. In emergency care, they can is a meaningful place such as somewhere with cul- accelerate access to medical records at the emergency tural significance, a shopping mall, a restaurant, or a bar [225]. site or seek urgent help from multiple experts virtually. In the surgical field, they can collect and process an 2.1.3. Environment Monitoring ever-increasing range of telemetric data from instru- Advances in sensing and wireless network technolo- ments used in an operating room and augment human gies provide realistic solutions for continuous moni- ability to detect patterns that could require immediate toring of natural environments. They potentially pro- action [24]. vide new data for environmental science as well as vi- To maintain the health and safety of the home and tal hazard warnings [83]. They allow the collection of residents and maximise the comfort of the residents, fine-grained environmental data, which is essential to researchers deploy sensors in a home setting to per- foster scientific research and to increase human un- ceive the state of the environment and its residences, derstanding of the environment. For example, detailed and reason about the state using artificial intelligence air pollution data collection helps to understand pol- techniques [49]. The DOMUS lab, at the University lutants’ spreading mechanisms and their influence on of Sherbrooke, achieves an implementation of smart human health [167]. In agriculture, sensors and actua- home based pervasive assistants which act as mobile tors operating at a much finer level of granularity can orthosis [50]. Its research goal is to collect ecologi- be used to precisely control, for example, the concen- cal data on symptoms and medication side-effects on tration of fertiliser in soil based on information gath- cognitively impaired people, establish their behaviour ered from the soil itself, the ambient temperature, and other environmental factors [76]. models, allow caregivers to monitor their daily activi- Pervasive sensing technologies are also particu- ties, and provide assistance adapting to different types larly important in remote, inhospitable, or danger- of cognitive deficits such as memory, initiation, plan- ous environments where fundamental processes have ning, and attention [25]. rarely been studied due to their inaccessibility [116]. 2.1.2. Transportation Some examples are: monitoring a chemical indus- Pervasive computing technologies are entering our trial environment to minimise the hazards of chemi- everyday life as embedded systems in transporta- cal wastage [109], monitoring gas emissions at land- tion [69,53]. A number of applications have emerged. fill sites [103], and studying active volcanoes with col- In tourist guides, a pervasive computing system can lected seismic and infrasonic (low-frequency acoustic) provide personalised services (like locating a particu- signals. lar type of restaurant or planning a day trip) for visi- 2.1.4. Summary tors based on their location and preferences [225]. In Beyond the above three types of applications, per- traffic control, a system can be immediately informed vasive computing has also been applied to other ar- of incidences of congestion or the occurrence of acci- eas, including workplaces, education, and office envi- dents and notify all approaching drivers. In route plan- ronments [50]. In education, intelligent classrooms are ning, a system can suggest the most convenient routes prototyped to enhance the classroom experience by au- for users based on the current traffic conditions and the tomating processes such as controlling light settings, transportation modes being used [223]. playing videos, and displaying slides [221]. Smart 4 Ye et alia / Semantically-enriched pervasive sensor-driven systems meeting applications facilitate archiving, analysing, 2.2.1. Summary and summarising meetings via a variety of technolo- We have briefly introduced the background of per- gies from physical capture from camera, microphone, vasive and sensor-driven computing in terms of appli- or other interactive tools and structural analysis to se- cations and research themes. From this, we conclude mantic processing [72]. that data and knowledge modelling is key in pervasive computing. We summarise the following requirements 2.2. Research Themes that are particularly relevant to the Semantic Web technology community. Pervasive sensing systems offer a framework for – Conceptual modelling: We require a commonly- new and exciting research across the spectrum of com- agreed vocabulary to represent data so as to make puter science. Major research themes cover the follow- them understandable across different systems and ing areas [188]: platforms; e.g., populating GPS or social data in – Sensing and reasoning In order to develop reac- Web communities, or transferring health records tive pervasive computing systems it is necessary across hospitals and medical organisations. Such to capture, process, and exploit the contextual pa- a vocabulary should be rich enough to represent rameters that inform and guide human behaviour. properties, structures, and relationships among – Privacy, trust, and security Privacy encompasses data, thus facilitating querying, personalising and reasoning about trust and risk involved in the in- further processing of data, as well as their sharing teractions between users and services. Trust con- and reuse. It will be used to semantically annotate trols the amount of information that can be re- data so not only are they represented in machine- vealed in an interaction. Risk analysis allows us to processable formats but they also are ‘meaning- evaluate the expected benefit that would motivate fully’ related to other data; e.g., in the GPS exam- users to participate in these interactions. Security ple in Section 2.1.2, a place needs to be annotated describes the cryptographic techniques used to se- not only as a location but also related to cultural cure the communication channels and participant or social aspects. data. – Reasoning: Reasoning is a key enabler for perva- – Discovery One of the key requirements for per- sive applications in that there is a large amount vasive computing systems is an approach or ser- of information in broad types that needs to be fil- vice capable of assimilating and filtering informa- tered and abstracted to higher-level information tion from its inputs (such as sensors, services, ap- that is understandable or interesting to applica- plications, and users). Given some infrastructure tions or humans; e.g., inferring a patient’s condi- to communicate this information, an approach to tion based on symptoms acquired from biochemi- matching that correlates relevant input events and cal sensors, or analysing air pollution data to pre- facts to a particular contextual service is needed. dict the pollution spreading direction so as to take This is essential to allow the user and the applica- immediate action to mitigate its effects. tion to discover the information from the environ- – Uncertainty: Context information is inherently ment necessary to achieve a defined goal or com- imperfect. Inaccuracies may easily arise, due to plete a task. erroneous and/or missing readings. Furthermore, – Interaction design As traditional computers dis- when pieces of context information come from appear from pervasive computing environments, multiple sources and modalities, ambiguities and novel human-computer interactions to handle the conflicts may also arise. Under these circum- peculiarities of their environments – including instances, modelling and reasoning need to provide visible devices, implicit interaction, and real, vir- the means to cope with such imperfections and tual, and hybrid interactions – need to be investi- allow to detect possible errors, handle gracefully gated. missing values, and derive plausible conclusions, – Essential infrastructure It is necessary to develop assessing the validity of available context data. tools to maintain and upgrade infrastructure in- – Service discovery: Pervasive and sensor-driven place over its lifecycle, as well as to allow the systems are usually assumed to operate in large- infrastructure to communicate failures effectively scale, open environments where components are to its users. loosely coupled and have volatile presence; i.e., Ye et alia / Semantically-enriched pervasive sensor-driven systems 5

they can enter or leave a system freely at any concept that describes a property of an environment point. Thus, it is desirable to associate them with or a user. For example, a location context can be semantic annotations so as to support matching, represented by means of the Coordinate, Geometri- discovery, composition, and orchestration in a cArea and SymbolicLocation classes. GeometricArea more intelligent manner. instances describe areas that map to a set of Coordi- – Privacy and trust: Data in pervasive systems are nate instances (numerical value array); SymbolicLoca- typically associated with privacy concerns; espe- tion instances assign a human-friendly symbol to Ge- cially personal data such as health records. We ometricArea instances. need a specification language to specify rules or Knowledge here indicates a corpus of knowledge policies about how to access, appropriately (per- specific to available context, including spatial knowl- haps automatically) abstract, and use data. edge for location contexts, or user preference and social network for person contexts. In Section 4, we introduce Semantic Web-based per- An event represents a change in the state of an envi- vasive systems and indicate which of these require- ronment that should be identified, processed and man- ments are favourably supported. In Section 5, we anal- aged by the system in order to deliver personalised ser- yse in detail the extent to which these requirements are vices to users; e.g., an event of ‘a user having heart at- satisfied. tack’ might trigger an application of calling an ambu- lance. A low-level event can be conceptually defined as 2.3. Information in Pervasive Sensing Systems an action or occurrence that happens at a certain time within an environment, which is described through ba- We introduce the types of information generated in sic elements such as when, where, what, and who.A a pervasive sensing system and highlight modelling high-level event includes other parameters including and reasoning challenges. Figure 1 lists information role and factor. including raw sensor data, events, context, domain Situations are defined as an abstraction of the states knowledge, and situations or human activities. occurring in the real world, which are usually inferred from the aggregation of multiple contexts. They may

High-level Event Human Activity Situation represent a human activity (e.g., a user is ‘having a group meeting’) or a state of affair (e.g., there is an ongoing presentation happening in a room). Situations, human activities, and high-level events are used inter- changeably in the community of pervasive computing, Low-level Event Context Knowledge depending on the application context. In the following, we will discuss the features of the above types of information in detail. Raw Sensor Data 2.3.1. Sensors and Sensor Data The service provision of a pervasive computing system relies on the perception of an environment, while Fig. 1. Different types of information in pervasive sensing systems the perception is accomplished by a range of sensors. Sensors are becoming smaller, lighter, lower in cost, Raw sensor data are readings reported from sensors; and aided by longer battery life. These advances in- e.g., a complete reading from Ubisense – a coordinate- creasingly support the embedding of sensors within based positioning sensor is ([4.54,1.92,4.70], an environment, integration with everyday objects, and tag_bob, 09/05/2008 09:49:39), including attachment to the human body. Sensors in pervasive when the reading is generated (e.g., 09/05/2008 computing can capture a broad range of information 09:49:39), who the reading is about (e.g., a tag be- on the following aspects [51]: longing to the user named bob), and what the reading value is (e.g., a 3D coordinate-based location datum – Environment: temperature, humidity, barometric [4.54,1.92,4.70]). pressure, light, and noise level in an ambient envi- Context is deﬁned by Dey et al. [57] as ‘any infor- ronment and usage of electricity, water, and gas; mation that can be used to characterise the situation – Device: state of devices (e.g., available or busy), of an entity’. We regard context as a well-structured functions of devices (e.g., printing or photocopy- 6 Ye et alia / Semantically-enriched pervasive sensor-driven systems

ing), the size of memory, the resolution of screen, 2.3.2. Context, Events and Situations or even operating systems embedded; Sensor data can be abstracted into a set of domain – User: location, schedule, motion data like accel- concepts, typically called context. Contexts can be eration of different parts of bodies, and biometric classified into different domains in terms of the prop- data like heart rate and blood pressure; erty they describe. Contexts in one domain can have a – Interaction: interactions with physical artefacts different structure, which is distinguishable from con- through RFID and object motion sensors [114], texts in other domains. For example in the domain of and interactions with devices through virtual sen- Location a coordinate context can be identified as three sors like monitoring the frequency a person uses numeric values with their unit of measurement as Me- their keyboard and mouse [133,123]. ter, while in the domain of Temperature a context can The diversity of sensors leads to high complexity be identified as a numeric value with its unit of mea- in their output, including volume, different modalities, surement as Celsius or Fahrenheit. A higher-level con- inter-dependence, real-time update, and critical age- text can be considered as a characteristic function on ing. In dealing with the real world, these sensors do sensor data; for example, a symbolic location context not always produce accurate data. Noisy sensor data studyRoom is mapped to a set of coordinate contexts may result in misunderstanding of a user’s or an envi- that are valid inside a study room. An event translates ronment’s state, which may lead to incorrect applica- or abstracts sensor data into a relation with contexts. tion behaviour. Sensors have technical limitations, are Semantic relationships exist between contexts in prone to breakdown, can become disconnected from one domain, including different levels of granularity, the sensor network, or may be vulnerable to environ- overlapping, and conflicting [218,170]. The seman- mental interference. This leads to issues surrounding tic relationships can be explored by applying domain the uncertainty of sensor data, which are out of date, knowledge or through a learning process. For example, incomplete, imprecise, and contradictory [84]. These given a map of a house, we can specify the spatial rela- features of pervasive sensor data can be an impediment tionships between the location contexts as: the location to the process of making them directly usable or un- context studyRoom is finer grained than another lo- derstandable by applications. cation context house (ref. spatially contained in), and Different sensors produce data of different modal- it conflicts with the context bedroom (ref. spatially ities, including binary, continuous numeric, and fea- disjoint with). tured values. The type of data has an impact on choos- Events are regarded as occurrences upon which ac- ing appropriate structure to represent it so as to make tions are taken. They are considered enduring entities it understandable and shareable among different plat- that unfold over time and space; that is, they occur or forms and systems. A binary value is the simplest type happen at certain time in certain space [168]. An event of sensor data: true (1) or false (0). RFID sensors pro- may be complex and have a variety of aspects beyond duce a binary reading: an object with a RFID tag is time and space, including objects and people involved, detected by a reader or not; binary-switch sensors be- and as well as mereological, causal, and correlative re- have similarly [205]. Continuous numeric values are a lationships between them [168]. common type of data that is produced by most sensors, A situation is defined as an external semantic in- including positioning sensors, accelerometers, and all terpretation of sensor data [217]. Interpretation means the ambient sensors. Featured values are typically pro- that situations assigns meanings to sensor data. Exter- duced from sophisticated sensors such as a camera and nal means that the interpretation is from the perspec- an eye movement tracker whose data needs to be char- tive of applications, rather than from sensors. Semantic acterised into a set of categorical measurements. For means that the interpretation assigns meaning on sen- example, motion features can be extracted from video sor data by defining structure or relationships within streams recorded in cameras, including quantity of mo- the same type of sensor data and between different tion and contraction index of the body, velocity, accel- types of sensor data. eration and fluidity [31]. Eye movements captured in A situation or human activity can be defined by col- electrooculography signals are characterised into two lecting relevant contexts, uncovering meaningful cor- types of features: saccades that refer to the simultane- relations between them, and labelling them with a de- ous movement of both eyes in the same direction and scriptive name. The descriptive name can be called fixations that are the static states of the eyes during a descriptive definition of a situation, which is about which gaze is held upon a specific location [26]. how a human being defines a state of affairs in reality. Ye et alia / Semantically-enriched pervasive sensor-driven systems 7

A logical expression of correlated context predicates 2.4. Summary is called a logical specification of a situation. With these two, a situation bridges sensor data and applica- As sensing technologies mature, pervasive comput- tions. Sensor data are abstracted to a certain situation ing is increasingly being deployed in real-world set- by evaluating its specification, and this situation will tings. To facilitate interoperability between sensors, trigger applications that correspond to its descriptive reasoners, and applications (that is, to support a shared name. understanding and a standard approach to query- There are rich structural and temporal relationships ing), we need a sensor model that provides machine- between situations: readable specifications of sensors, their output types, and the domains in which they operate so that it can Temporal feature The occurrence of a situation usu- support classifying sensors according to their function- ally exhibit certain temporal features: time-of-day alities, finding a sensor that can perform a certain mea- – a situation may only happen at a particular time surement, or even composing existing sensors to create of the day; duration – it may only last a certain a virtual sensor for particular functionality [48]. The length of time; frequency – it may only happen a sensor model also needs to capture information aspects certain number of times per week, and sequence about observation data such as when, what and qual- – different situations may occur in a certain se- ity measures (e.g., sampling frequency, coverage area, quence. A situation may occur before, or after precision or accuracy), as well as to enable the derivation of high-level knowledge from low-level sensor another situation, or interleave with another sit- data. uation; for example, ‘taking pill’ should be per- The explicit modelling of events is beginning to gain formed after ‘having dinner’ [97]. widespread interest, as the increasing number of sen- Generalisation A situation can be regarded as more sors leads to an ubiquity of events that need to be general than another situation, if the occurrence recognised and communicated. Event detection, clus- of the latter implies that of the former; for ex- tering and annotation are the major concerns in event ample, a ‘watching TV’ situation is considered modelling. more specific than an ‘entertainment’ situation, Context- and situation-awareness is a key enabler of because the conditions inherent in the former sit- pervasive systems; that is, the behaviour of the services uation subsume or imply the conditions in the lat- provided, depends on a system’s ability to respond to ter situation [216]. users’ current contexts or situations. We need a context Composition A situation can be decomposed into a and situation model with rich semantics to express un- set of smaller situations, which is a typical com- certainty, temporal features, generalisation, composi- position relation between situations. For exam- tion, and dependence. Such a model also requires pow- ple, a ‘cooking’ situation is composed of a ‘us- erful reasoning capabilities to effectively represent and ing stove’ situation and a ‘retrieving ingredients’ efficiently reason on these properties and relationships. situation. McCowan et al. propose a two-layered framework of situations: a group situation (e.g., ‘discussion’ or ‘presentation’) is defined as a 3. Semantic Web Technologies composition of situations of individual user (e.g., ‘writing’ or ‘speaking’) [119]. The Semantic Web [14] is a resource-oriented extension of the current Web that aims to afford a com- Dependence A situation depends on another situa- mon framework for sharing and reusing data across tion if the occurrence of the former situation is heterogeneous agents, applications and systems. The determined by the occurrence of the latter situ- underlying rationale is to make the semantics of web ation. Dependence can be long- or short-range, resources explicit by attaching to them metadata that as proposed by [42]. Sometimes long-range de- describe meaning in a formal, machine-understandable pendence can be more useful in inferring high- way. Within this vision, ontologies play a key role, pro- level situations. For example, a situation ‘going to viding consensual and precisely defined terms for the work’ may be better in inferring a situation ‘go- description of resources in an unambiguous manner. ing home from work’ than other short-range de- In 2004, the Web Ontology Language (OWL) [204] pendent situations. became a W3C recommendation, paving the way 8 Ye et alia / Semantically-enriched pervasive sensor-driven systems for the development of adequate tool support (on- Table 1 tology editors, reasoners, etc.) and the proliferation Terminological and assertional axioms of ontology-based applications in a plethora of do- Name Syntax Semantics mains. Formally founded in Description Logics (DLs) Concept inclusion C v D CI ⊆ DI [9], OWL is endowed with expressive representational Concept equality C ≡ D CI = DI constructs that allow to capture complex knowledge. Role equality R ≡ S RI ⊆ SI At the same time, OWL avails of the well-defined DL Role inclusion R v S RI ⊆ SI reasoning services for affording automated reasoning Concept assertion C(a) aI ∈ CI support. Role assertion R(a, b) (aI , bI ) ∈ RI The aforementioned furnish OWL with a multiplic- ity of appealing, within the context of pervasive appli- the concept Person; intuitively this corresponds to all cation features. For example, in OWL one can effec- those individuals that are friends with at least one per- tively model and reason over taxonomic knowledge. son. This is a desirable feature in pervasive applications A DL knowledge base K typically consists of a where the need to model information at different lev- TBox T (terminological knowledge) and an ABox A els of granularity and abstraction, so as to drive the (assertional knowledge). The TBox contains axioms derivation of successively further detailed contexts is that capture the possible ways in which objects of a do- particularly evident. Similarly, OWL supports consis- main can be associated. For example, the TBox axiom tency checking, another useful feature when dealing Dog v Animal asserts that all objects that belong to with imperfect context information coming from mul- the concept Dog, are members of the concept Animal tiple sources. too. The ABox contains axioms that describe the real- Last but not least, OWL adheres to the open-world world entities through concept and role assertions. For assumption, providing an elegant way of modelling in- example, Dog(Jack) and isLocated(Jack,kitchen) ex- complete information. Intuitively, open-world seman- press that Jack is a dog and he is located in the kitchen. tics assumes that we do not have complete informa- Table 1 summarises the set of terminological and as- tion about the world, but that some information may sertional axioms. be missing. Such assumption is well-suited for sensor- The semantics of a DL language is formally de- driven systems, where information may be incomplete fined through an interpretation I that consists of a non- due to a number of reasons, such as sensor inaccu- empty set ∆I (the domain of interpretation) and an in- racies or imperfect observations as described in Sec- terpretation function .I , which assigns to every atomic tion 2.3.1. concept A a set AI ⊆ ∆I and to every atomic role R a binary relation RI ⊆ ∆I × ∆I . The interpretation of 3.1. Description Logics complex concepts follows inductively. Table 2 shows the syntax and semantics of some of the most common The design of OWL, and particularly the formalisa- DL constructors. tion of the semantics and the choice of language constructors, have been strongly influenced by Descrip- 3.2. Reasoning Services tion Logics (DLs) [93]. DLs are a family of knowledge representation formalisms characterised by logi- Besides formal semantics, DLs come with a set cally grounded semantics and well-defined reasoning of powerful reasoning services, for which efficient, services [9]. sound and complete, reasoning algorithms, with well- The main building blocks are concepts represent- understood computational properties, are available ing sets of objects (e.g., Person), roles representing (e.g., tableaux-based algorithms [10]). Example state- relationships between objects (e.g., worksIn), and in- of-the-art implementations include Pellet [174], Racer dividuals representing specific objects (e.g., Alice). [82], Fact++ [200] and Hermit [132]. Starting from atomic concepts, such as Person, ar- Assuming a DL knowledge base K = (T,A), typical bitrary complex concepts can be described through reasoning services include: a rich set of constructors that define the conditions on concept membership. For example, the concept – Subsumption: A concept C is subsumed by D in ∃hasFriend.Person describes those objects that are re- T (written C v D), iff CI ⊆ DI for all interpre- lated through the hasFriend role with an object from tations I. Ye et alia / Semantically-enriched pervasive sensor-driven systems 9

Table 2 presence of both OccupiedRoom(kitchen) and Empty- Examples of concept and role constructors Room(kitchen) leads to inconsistency. Name Syntax Semantics Instance checking denotes the task of finding whether Top > ∆I a specific individual is an instance of a given concept. Bottom ⊥ ∅ Realisation of an individual, a more generic form of Intersection C u D CI ∩ DI instance checking, returns all (most specific) concepts Union C t D CI ∪ DI from the knowledge base that a given individual is an Negation ¬C ∆I \ CI instance of. Its dual is the retrieval problem that given {a ∈ ∆I | ∀b.(a, b) ∈ RI a specific concept C it returns all individuals that be- Universal quant. ∀R.C → b ∈ CI } long to this concept. This reasoning service is the cen- {a ∈ ∆I | ∃b.(a, b) ∈ RI tral one to realise the task of recognition of situation Existential quant. ∃R.C ∧ b ∈ CI } types. {(b, a) ∈ ∆I × ∆I | A particularly appealing feature of DLs is that they Inverse R− (a, b) ∈ RI } are decidable fragments of first-order logic. Their de- Transitive closure R+ S (RI )n cidability is primarily due to the so called tree model n≥1 property [206], according to which a class C has a Composition R ◦ S RI ◦ SI model (i.e. an interpretation I in which CI is non- empty) iff C has an interpretation that defines a tree- Equivalence: Two concepts C and D are equiva- – shaped directed graph. As we later discuss, this prop- lent in T (written C ≡ D) iff C v D and D v C. erty has a direct impact on the relational expressivity of Disjoint: A concept C is disjoint to a concept D in – DLs and OWL, and has been among the principal mo- T iff in every interpretation I it holds that CI 6= tivating factors for investigating the integration with ∅. rules. Satisfiability: A concept C is satisfiable in T iff – Falling under the Classical logics paradigm [144], there is an interpretation I such that CI 6= ∅. reasoning in DLs (and hence in OWL) adopts the – Consistency: The ABox A is consistent w.r.t. T iff open-world assumption. Intuitively, if a fact a holds if there is an interpretation that is a model of both only in a subset of the models of the knowledge base A and T . KB, then we can conclude neither KB |= a nor KB – Instance checking: The individual a is an instance 6|= ¬a. For example, if the only available knowl- of C w.r.t. (K) (written K |= C(a)) iff aI ∈ CI edge regarding the residents of a house is the asser- holds for all interpretations I of K tion livesIn(Alice,house), we cannot deduce based on – Realisation: An instance’s a realisation w.r.t. to it alone that no one else lives in the house. In con- K includes finding the most specific concepts C trast, formalisms adhering to the closed-world assump- for which aI ∈ CI holds for all interpretations I tion make the common-sense conjecture that all rele- of K. vant information is explicitly known, so all unprovable Hence, through subsumption one can derive the im- facts should be assumed not to hold. In our example, plicit taxonomic relations among the concepts of a ter- this amounts to concluding that Alice is the sole resi- minology. For example, given the axiom Occupied- dent of this house. Hence, closed-world reasoning can Room v Room u ∃contains.Person, one can infer that be intuitively understood as reasoning where from KB Room subsumes OccupiedRoom. 6|= a one concludes KB |= ¬a [129]. Such kind rea- Satisfiability and consistency checking are useful to soning should not be confused however with closed- determine whether a knowledge base is meaningful at domain reasoning, which has also been used in a num- all. Satisfiability checking enables the identification of ber of context reasoning frameworks (Section 5.4), and concepts for which it is impossible to have members which involves reasoning only over explicitly known under any interpretation (for example, an unsatisfiable individuals. concept, though trivial, is OccupiedRoom u ¬ Occu- piedRoom). Consistency checking enables the identi- 3.3. OWL and OWL 2 fication whether the set of assertions comprising the knowledge base is admissible with respect to the termi- OWL comes in three dialects of increasing expres- nological axioms. For example if EmptyRoom and Oc- sive power: OWL Lite, OWL DL and OWL Full [7,91]. cupiedRoom are asserted as disjoint concepts, then the The first two languages can be considered as syntactic 10 Ye et alia / Semantically-enriched pervasive sensor-driven systems variants of the SHIF(D) and SHOIN (D) DLs re- as this leads to a cyclic dependency. Three profiles, spectively. The third language was designed to provide namely OWL 2 EL, OWL 2 QL and OWL 2 RL, trade full compatibility with RDF(S), which is the most ex- portions of expressive power for efficiency of reason- pressive of the three. It neither imposes any constraints ing targeting different application scenarios. on the use of OWL constructs, nor lifts the distinction between instances (individuals), properties (roles) and 3.4. Combining Ontologies and Rules classes (concepts). This high degree of expressiveness comes however at a price, namely the loss of decid- To achieve decidability DLs, and hence OWL, trade ability that makes the language difficult to implement. some expressiveness for efficiency of reasoning. The As a result, focus has placed on the two decidable di- tree-model property, mentioned above, is one such ex- alects, and particularly on OWL DL, which is the most ample. It conditions the tree-shape structure of models, expressive of the two. ensuring decidability, but at the same time it severely Despite the rich primitives provided for expressing restricts the way variables and quantifiers can be used, concepts, OWL DL has often proven insufficient to ad- dictating that a quantified variable must occur in a dress the needs of practical applications. This limita- property predicate along with the free variable. As a tion amounts to the Description Logic style model the- result, it is not possible to describe classes whose in- ory used to formalise its semantics, and the tree-model stances are related to an anonymous individual through property role in conditioning DLs decidability. As a different property paths [78]. To leverage OWL’s lim- consequence, OWL can model only domains where ited relational expressivity and overcome modelling objects are connected in a tree-like manner. This con- shortcomings that OWL alone would be insufficient to straint can be restrictive for real-world applications, address, a significant body of research has been de- including the pervasive domain, which requires mod- voted to the integration of OWL with rules. elling general relational structures (see Section 5.4). A proposal towards this direction is the Semantic Responding to this limitation and to other draw- Web Rule Language (SWRL) [92], in which rules are backs that have been identified concerning the use of interpreted under the classical first order logic seman- OWL in different application contexts throughout the tics. Allowing concept and role predicates to occur in years, the W3C working group produced OWL 2 [77]. the head and the body of a rule without any restrictions, OWL 2 (equivalent to the SROIQ(D) DL) is a re- SWRL maximises the interaction between the OWL vised extension of OWL, now commonly refereed to and rule components, but at the same time renders as OWL 1. It extends OWL 1 with qualified cardinal- the combination undecidable. To regain decidability, ity restrictions; hence one can assert that a social ac- proposals have explored syntactic restrictions on rules tivity is an activity that has more than one actors: So- [162,131] as well as their expressive intersection [78]. cialActivity v Activity u ≥ 2 hasPar- The DL-safe rules introduced for example in [131] im- ticipant.Person. Furthermore, it makes it possi- pose that rule semantics apply only over known indi- ble to define properties to be reflexive, irreflexive, tran- viduals. It is worth noting that in practice DL reason- sitive, and asymmetric, and to define disjoint pairs of ers providing support for SWRL actually implement a properties, thus providing extended support for captur- subset of SWRL based on this notion of DL-safety. ing mereology relations. Parallel to these efforts, a highly challenging and Another prominent OWL 2 feature is the extended active research area in the Semantic Web addresses relational expressivity that is provided through the the seamless integration of open and closed world se- introduction of complex property inclusion axioms mantics. Representative initiatives in this quest include (property chains). To maintain decidability, a regular- among others the hybrid formalism of MKNF knowl- ity restriction is imposed on such axioms that dis- edge bases proposed by Motik and Rosati [130], the allow the definition of properties in a cyclic way. extension of ontologies through the use of integrity Hence, one can assert the inclusion axiom locatedIn constraints proposed by Tao et al. [190], and the so- ◦ containedIn v locatedIn making it possible to in- called grounded circumscription approach [108]. fer that if a person is located for example in the En- Taking a different perspective, a number of ap- gineering Department of the University, then she is proaches have investigated the combination of ontolo- located in the University as well; however, it is not gies and rules based on mappings of a subset of the on- allowed to use both the aforementioned axiom and tology semantics on rule engines. For instance, Horst the axiom containedIn ◦ locatedIn v containedIn [193] defines the pD* semantics as a weakened variant Ye et alia / Semantically-enriched pervasive sensor-driven systems 11 of OWL Full, e.g., classes can be also instances, and 4.1. SOUPA in [194] they are extended to apply to a larger subset of the OWL vocabulary, using 23 entailments and 2 in- The Context Broker Architecture (CoBrA in Fig- consistency rules. Inspired by the pD* entailments and ure 2), is a broker-centric, agent-based architecture DLP [78], the semantics of the OWL 2 RL profile is supporting context-aware computing in intelligent realised as a partial axiomatisation of the OWL 2 se- spaces [102]. CoBrA provides a means of acquir- mantics in the form of first-order implications, known ing, maintaining, and reasoning about context; shar- as OWL 2 RL/RDF rules. User-defined rules on top ing knowledge; detecting and resolving inconsistent of the ontology allow to express richer semantic re- knowledge; and protecting user privacy [35]. All of the lations that lie beyond OWL’s expressive capabilities, above capabilities are provided by the context broker, and couple ontological and rule knowledge. Examples an intelligent agent that is the central component in of rule-based OWL reasoners that implement entail- CoBrA. Inside the context broker, the Context Acqui- ment rules, include among others OWLIM [18], OWL- sition Module offers a library of procedures that form JessKB1, Jena [117], O-DEVICE [124] and DLEJena a middleware abstraction for context acquisition; the [125]. Context Knowledge Base maintains a shared model of context on the behalf of a community of agents and de- 3.5. Summary vices in the smart space; the Context Reasoning Engine reasons over the context so as to detect and resolve the This section introduced the basic notions underly- inconsistent knowledge; and the Policy Management ing the Semantic Web and provided a brief overview Module provides a set of user-defined inference rules of key technologies empowering the envisaged knowl- that deduce instructions for deciding the right permis- edge sharing and reuse across heterogeneous environ- sions for different computing entities to share a par- ments. Expressive ontology languages allow to ele- ticular piece of contextual information, and for select- gantly capture complex knowledge and its semantics ing the recipients to receive notifications of context in a formal way, rendering it amenable to automated changes. reasoning tasks with well-understood computational Contextual information in CoBrA is represented by properties. Rules augment further the expressive ca- a set of ontologies, called COBRA-ONT that is im- pabilities, by allowing to represent richer semantic re- plemented in OWL. COBRA-ONT is the key require- lationships. Given the inherently open nature of per- ment for modelling context in the smart meeting ap- vasive, sensor-driven systems, where the need to inte- plication [32]. It defines typical concepts and relations grate domain knowledge and meaningfully aggregate for describing physical locations, time, people, soft- and interpret low-level context information is a cru- ware agents, mobile devices and meeting events. A cial requirement, it comes as no surprise that Semantic set of more general ontologies, named SOUPA (Stan- Web technologies have been acknowledged as afford- dard Ontology for Ubiquitous and Pervasive Appli- ing a number of highly desirable features. cations), that encompasses COBRA-ONT, has been proposed for supporting pervasive computing applications. SOUPA [37] borrows terms from other stan- 4. Ontology-modelled Pervasive Systems dard domain ontologies such as FOAF [65], DAML- Time [88], OpenCyc [111], RCC [23,46], and the Rei This section provides an overview of how ontolo- Policy [101] Ontology. gies are employed in a pervasive system by introduc- SOUPA offers a formal, well-structured way to ing representative ontology models in the area. Gen- model context and thus provides rich semantics for erally, there are two main functions of ontologies in programming. It also allows policies to be defined to pervasive computing: semantic enrichment at a con- support trust and privacy. This is demonstrated in Co- ceptual level and support for developing knowledge- BrA’s EasyMeeting application [34], in which the on- centric software [39]. The following examples are the tologies facilitate knowledge sharing and work with most influential models and systems that demonstrate logic inference rules to reason about context. SOUPA these two functions. also applies the MoGATU ontology to express the beliefs, preferences, intentions, and desires of an agent 1http://edge.cs.drexel.edu/assemblies/ or a user, which makes it possible to rank the priorities software/owljesskb/ of plans and goals. 12 Ye et alia / Semantically-enriched pervasive sensor-driven systems

Information Servers Semantic Web & (Exchange Server, iCal, Web Services YahooGroups, etc) (RDF, DAML+OIL & OWL) Database (MySQL) Contexts in External Sources

Context-Aware ContextBroker Agents Context Knowledge base Meeting Context Agent Context SOAP + RDF/OWL -Aware Reasoning Engine RIPA-ACL + RDF/OWL Personal Presentation Agent Devices Context Rec Agent Acquisition Module Lunch Emergency Privacy Agent Alert Management Module Agent

Smart Tag Sensors Environment Sensors Device & Gadget Sensors (RFID) (Xanboo & X10 technology) (Java Ring, SmartCart etc)

Fig. 2. CoBrA – an intelligent context broker (source: [38])

4.2. Gaia ioms and constraints on the properties of the entities that must be satisfied. However, Gaia does not define Gaia is an infrastructure for smart spaces, which specific ontologies for each of the core concepts of per- are pervasive computing environments that encompass vasive computing. For example, the location context is physical spaces [161]. The main characteristic of Gaia represented in predicate form, where the location pred- is that it brings the functionality of an operating system icate must have a subject which belongs to the set of to physical spaces. It employs common operation system functions (including events, signals, file systems, persons or things, a verb or preposition like ‘inside’ security, and processes), and extends them with con- or ‘entering’, and a location, which may be a room or text, location awareness, mobile computing devices a building, e.g., Location(chris, entering, and actuators. Using this functionality, Gaia integrates room 3231). the devices and physical spaces, and allows the physi- In Gaia, the Ontology Server is responsible for load- cal and virtual entities to seamlessly interact. ing and validating ontologies from DAML+OIL docu- Ontologies are introduced in Gaia as an efficient ments, and composing ontologies into a combined on- way to manage the diversity and complexity of describing the resources (e.g., devices and services). First tology for the entire system. It is also capable of serv- of all, they work as a system specification for configu- ing logical queries to the Knowledge Base that repre- ration management by providing a standard machine- sents the dynamically integrated ontology base. The readable taxonomy of the different kinds of entities, in- Ontology Explorer is a graphical user interface (GUI) cluding applications, services, devices, users, and data that allows users to browse and search ontologies, and sources. Therefore, these ontologies are beneficial for to interact with other entities in the space [121]. The semantic discovery [120], matchmaking [199], inter- ontology explorer offers a clearer understanding of the operability between entities [122], and interaction be- semantics of a system. This also makes it easier to tween human users and computers [157]. Gaia ontologies [156] model contextual informa- write rules that determine context-sensitive behaviour. tion including physical, environmental, personal, so- The explorer helps human users to interact directly cial, application, and system contexts. They describe with the constituent parts of a pervasive computing en- relations between different entities and establish ax- vironment. Ye et alia / Semantically-enriched pervasive sensor-driven systems 13

4.3. ASC Context Application Context-aware Service Context-aware Service Layer Aspect-Scale-Context (ASC) is a contextually based Context model [186]. A context is a set of contextual informa- Reasoner Service Locating Context tion characterising entities (like a person, place, or a Service Database Context KB general object) relevant for a speciﬁc task in their rel- Context Interpreter Context Middleware evant aspects. An aspect is a classiﬁcation, symbol- or Layer External Context Internal Context value-range, whose subsets are a super-set of all reach- Provider Provider able states, grouped in one or more related dimensions

Context called scales [187]. The ASC model shows how con- Virtual Sensors Sensing (Web Sensors, Physical Sensors Layer textual information is used to characterise a state of an Information Servers) entity under a specific aspect. Ontologies implemented in ASC serve as the un- Fig. 3. Architecture of SOCAM (source: [79]) derlying basis for describing contextual facts and in- terrelationships, which improve the formality of the The CONtext ONtology (CONON) is an ontology- model. They also facilitate service discovery and ser- based context model, in which a hierarchical approach vice interoperability on the context level. CoOL, the is adopted for designing context ontologies [80]. The Context Onotology Language [187] is derived from ontologies include a common upper ontology for the ASC to facilitate ontology-based contextual interoper- general concepts in pervasive computing (such as ability. CoOL is divided into two subsets: the CoOL person, location, computing entity, and activity) and Core, which projects the ASC model into common on- domain-specific ontologies that apply to different sub- tology languages such as OWL and DAML+OIL, and domains like smart homes. The context model supports F-Logic [104]; and CoOL Integration, which is a col- multiple semantic contextual representations like con- lection of schema and protocol extensions as well as textual classification, contextual dependency and qual- common sub-concepts of ASC. OntoBroker [56], the ity of context. Using CONON, two types of contex- inference engine, deals with knowledge query issues tual reasoning tasks are supported: ontology reason- by specifying knowledge as conclusions from ontolo- ing with DLs, and user-defined reasoning by defining gies. The advantage of CoOL is that it enables context specific rules in first-order logic. interoperability and context-awareness during service The CONON ontologies help sharing common un- discovery and execution. derstanding of the structure of contextual information [187] describes an overall architecture of the sys- from users, devices and services so as to support se- tem that focuses on the context provider domain. The mantic interoperability and reuse of domain knowl- main parts of this middleware component include on- edge. They also support efficient reasoning mecha- tologies, facts, rules, and the inference engine. The first nisms so as to check the consistency of context and de- two are CoOL based knowledge, and with given rules duce higher-level, implicit context from raw context. they deliver knowledge to the inference engine. CoOL A context-aware home scenario is implemented in the is capable of deriving new knowledge, validating con- prototype system to demonstrate the work of ontolo- sistency and asserting inter-ontology relationships so gies [80]. as to integrate ontologies. 4.5. Ontonym 4.4. SOCAM Ontonym [177] comprises a set of upper ontolo- The Service-Oriented Context-Aware Middleware gies [12] and a methodology for constructing high- (SOCAM in Figure 3) is an architecture which enables level pervasive environment descriptions. Rather than the building and rapid prototyping of context-aware support the complete specification of a pervasive sys- services in pervasive computing environments [80]. tem, Ontonym aims to provide a basis for semantic The middleware abstracts physical spaces from which interoperability across pervasive applications and en- contexts are acquired into a semantic space where con- vironment by providing a core set of concepts from texts can be easily shared and accessed by context- which application-level concepts can be defined by aware services. means of extension and composition. 14 Ye et alia / Semantically-enriched pervasive sensor-driven systems

Beyond commonly recurring concepts such as peo- Reusable Third-Party Application Advanced Interaction ple, location, events, and resources, the core offerings Services Apps & Tools Layer Monitoring & Directing of Ontonym include: a model of time that supports the specification of historical and predicted states; a sensing model that captures properties of data sources such Intelligent Processing Layer isualisation as their operational tolerances, capability, and sens- Aggregation Learning Mining Portal V Provenance ing frequency; a model of provenance for recording sources of data and the role of any intermediate com- RDF Data Bus (RDF+OWL+HTTP+SPARQL) ponents in its derivation [27]; and support for capturing imprecise information by means of an uncertainty vocabulary. Semantic Layer Ontonym has a number of strengths. In particular, it SH1, SH2 in institution 1 SH1, SH2 in institution 2 is designed with orthogonality in mind, with the concepts in any two given ontologies loosely coupled in Data Layer order to support reuse. Particular attention is paid to Data Flat Files Database 1 reusing and linking to terms from existing standard vocabularies (e.g., OWL-Time [141] and the Measure- Multimedia 2 Actuators Sensor 1 Physical Layer Sensor 2 ment Units Vocabulary ), avoiding encoding bias; that Outputs is, information is not represented for the convenience of particular applications. Ontonym’s Person and Lo- Fig. 4. The conceptual architecture of the SSH (source: [39]) cation vocabularies are more expressive than the offerings of other pervasive computing vocabularies, and it is the only pervasive computing vocabulary to consider sharing, integration and fusion across multiple organ- the representation of sensors and (basic) provenance isations. Underlying the semantic layer are the smart information. home ontologies for representing and reasoning on data, including low-level sensor data, domain knowl- 4.6. Semantic Smart Home edge, and high-level human activities. Among the smart home ontologies, there is an activ- Semantic Smart Home (SSH) is considered as an ex- ity ontology that models human activities as a hierar- tension of the current smart home solutions with se- chy of classes with each class described by a number mantic dimension, which will enable semantic-based of properties. The engineered activity models capture knowledge discovery and intelligent processing. The built-in interrelations between an activity and objects essence of a SSH is that data, devices, and services are that are used when a user is performing the activity. given well-defined meaning so that they can be linked For example, the activity MakeTea is a subclass of within and across smart homes. The semantic exten- MakeHotDrink, and hasDrinkType is a property sion makes it possible to exploit advanced semantic of MakeHotDrink, which links the class Drink- or artificial intelligence based information processing Type (e.g., its instances are tea, coffee, or chocolate) techniques to provide value-added data processing ca- to MakeHotDrink. The process of activity recogni- pabilities such as data integration, interoperability and tion is to perform equivalency and subsumption rea- high-level decision support [39]. soning to check whether a conceptual description of an SSH is realised in a layered conceptual architecture, activity formed by collected sensor data is equivalent consisting of the physical layer that hosts a number to any atomic activity concept [41]. of sensors or devices, the data layer that collects raw data from sensors, the application layer that provides 4.7. Top-level Ontologies in Smart Home assisted living applications, and the semantic layer be- Environments tween the data and application layers. The semantic layer aims to provide a homogeneous view over het- Most ontology models in the above systems follow erogeneous data, thus enabling seamless data access, the philosophical nature of ontology; that is, defining a commonly agreed vocabulary for representing data 2http://purl.oclc.org/NET/muo/muo and knowledge structure. The vocabulary makes data Ye et alia / Semantically-enriched pervasive sensor-driven systems 15 understandable, sharable, and reusable by both humans troduce additional systems and ontologies beyond the and machines. However, developing such ontologies above approaches. usually takes a large amount of knowledge engineering In terms of the Semantic Web technologies intro- effort from developers. duced in Section 3, OWL DL is the language used most Ye et al. design and develop a top-level ontology by the afore-described systems. They are usually con- model [219], which captures essential semantics com- figured through multiple reasoners, among which the mon to instances of all dimensions of an informa- most used ontological reasoners are Racer, FaCT and tion space; i.e., levels of granularity, conflicting, and Pellet. As yet, OWL 2 profiles have not been notably overlapping relationships. For example in Temperature employed. dimension, an instance hot is defined on the range [30, ∞], and another instance cold on [−∞, 0]. We can infer that hot is conceptually conflicting with 5. Integrating Semantic Web with Pervasive cold. Meta-properties are explored on these three re- Systems lationships, which facilitate deriving new knowledge from a limited amount of specified knowledge. From The aforementioned outlined key Semantic Web the individual instances, context predicates and activ- features and provided a first overview of ontology- ities are constructed. A context predicate is defined as based pervasive systems. In the following we explore a two-dimensional product space of information, and in detail how these features meet the requirements of an activity is a multiple-dimensional product space of typical pervasive systems, including the representation information that indicates a situation occurring in the sensor data, the modelling and querying of context environment. Generic reasoning rules are specified to and events, context-driven reasoning, addressing un- derive relationships between context predicates and re- certainty, semantic service discovery, privacy and trust. lationships between activities. The derived relation- 5.1. Representing Sensor Data ships facilitate system-level tasks in pervasive computing, including detecting contradictory sensor events, Hossein et al. propose a Sensory Data Set Descrip- checking consistency of activity rules, and aiding ap- tion Language (SDDL), which is an XML-encoded plication design on an inferred hierarchy of activities. description language for sensor data originating from Schmidtke et al [170] also propose a top-level on- pervasive spaces [94]. The scope of SDDL is to specify tology, which is built on the theory of mereotopology. information about the pervasive space, including: (1) Mereotopology is a formal theory of part and whole available sensors/actuators such as the sensor id, mea- that has been widely used in the area of formal on- surement unit, its type, and where it is installed, (2) tology. From a primitive relation part-hood, they dis- data set parameters such as the space layout of the en- cuss the overlap, underlap, and connection relations vironment, the project specification, and contact infor- between contexts. Two contexts overlap if and only if mation, and (3) sensor events such as the timestamp, they share a part that is not identical to the empty (i.e., activated sensor id and its value, and whether or not it non-existing) context, while two contexts underlap if is noisy. It does not concern physical properties of the and only if they are both part of a context that is not sensors. The motivation behind this work is to provide identical to the whole domain. A connection can be a standard description language for datasets so as to introduced between two contexts, which is a reflexive facilitate sharing the existing and future datasets. and symmetric relation. These relations are considered Built on SDDL, Ye et al. propose a generic con- crucial in modelling the notions of reachability in a ceptual model PI to model sensors and their quality conceptual hierarchy. measure for existing smart home data sets [220]. Each sensor type has a set of properties including its name, 4.8. Summary id, domain – a parameter that describes what the sensor type measures (e.g., acceleration or usage of gas), Table 3 shows that of the requirements introduced in and a unitOfMeasurement – the unit used to report ob- Section 2.2.1,the above ontology models mainly focus served values. Each sensor type can be associated with on conceptual modelling and reasoning, while the ser- a technical specification that includes its manufacturer, vice discovery and privacy and trust are added values model, size, deviation of readings produced by its sen- to the semantic modelling. The following sections in- sors (e.g., ‘a maximum precision of ± 1.5% full scale’ 16 Ye et alia / Semantically-enriched pervasive sensor-driven systems

Table 3 Evaluation of requirements in pervasive computing satisﬁed by ontology-based models in Section 4

Ontology-based Systems Conceptual modelling Reasoning Service discovery Privacy and trust √ √ √ CoBrA (SOUPA) – √ √ √ √ Gaia √ √ √ ASC (CoOL) – √ √ √ SOCAM (CONON) – √ √ Ontonym –– √ √ Semantic smart home –– √ √ Top-level ontologies ––

Observation Sensor sorML is a syntactic model for sensors, while the se- Object sensorId id 1 1 id value 1..n name 1 mantic issues such as interpreting and integrating its location assertedAt typeId function validDuring object encoded information have not been fully resolved [48]. * Based on SensorML, Russomanno et al. propose a 1 TechSpecification SensorType model comprehensive deep sensor ontology [164], called On- id size 0..1 name 1 valueRange toSensor. OntoSensor also consists of other standardi- unitOfMeasurement manufacturer domain frequency sation such as the IEEE Suggested Upper Merged On- deviation tology (SUMO) [189] and ISO 19115 [96]. OntoSen- sor includes a domain theory expressed in a language Fig. 5. The sensor and quality specification model in PI (source: that is constructed using the functional and relational [220]) basis sets to support ontology-driven inference. It is able to capture sensors’ computing and communica- for a gas flow sensor), its sampling frequency, and tion capabilities and their percept attributes that are a number of valueRange parameters – boundary val- used to store measurements of physical phenomena, ues that characterise different states. These values may and detection, classification and tracking of physical come from the manufacturer or through the applica- objects. For example, a thermal sensor is associated tion of learning techniques. It supports deriving high- with an attribute sample_interval on which a level context from raw sensor readings; e.g., a context processing service detectAlarm(threshold) is stoveOn is inferred if the gas sensor on the stove has defined; that is, if the thermal expansion or contraction a reading greater than a threshold. at regular sample intervals are beyond a threshold, then an alarm is reported. OntoSensor also includes more [stoveONRule: advanced inference mechanisms that can be used for (?A sensor:domain "GAS"), synergistic fusion of heterogeneous data. (?A sensor:value ?B), (?A sensor:object ?C), More sensor ontologies can be found in a review (?C rdf:type :Stove) done by the W3C [209] and Compton et al [48]. ge(?B "1"^^xsd:double) Based on the review, the W3C Semantic Sensor Net- -> createClassification(?A, "stoveOn")] work Incubator group [208] has developed the Seman- tic Sensor Network (SSN) ontology [47]. It targets at SensorML, the Sensor Model Language, is specified the formal and machine-processable representation of as a generic data model in UML for capturing classes sensor capabilities, properties, observations and mea- and associations that are common to all sensors. It is surement processes, with which it aids in searching, part of an Open Geospatial Consortium (OGC) ini- querying, and managing the sensor network and its tiative to contribute to the development of a Sensor data. Central to the ontology is the Stimulus-Sensor- Web [165,164]. SensorML provides an XML encoding Observation (SSO) ontology design pattern [98] that schema for developing a persistent data store for sensor provides a lightweight model for representing sensors, metadata and sensed attribute values. Chu et al. apply their inputs (called stimulus) and observations. SSO it to model web-hosted sensors to enable remote dis- is reusable for a variety of application areas and it covery, access and usage of real-time data [44]. Sen- can be used in conjunction with other relevant ontolo- Ye et alia / Semantically-enriched pervasive sensor-driven systems 17 gies. Both SSN and SSO have been aligned with the 5.2.1. When—Time DOLCE+DnS Ultralite (DUL) ontology [64] so as to Temporal features play a key role in enhancing en- enable the integration into more complex ontologies tity descriptions within a pervasive environment. For as a common ground for alignment, matching, transla- example, representing the time at which an event takes tion, or interoperability. place, the frequency with which a sensor samples the environment (e.g., every 10 seconds), or the expected 5.1.1. Analysis duration of an activity (e.g., 30 minutes). Here, we The above sensor-centric and observation-centric must also consider semantically meaningful notions of ontologies have developed from a syntactic model to a time, which may have an exact (e.g., Monday morn- semantic and knowledge model. SDDL and PI tend to ing, Christmas) or fuzzy (e.g., lunch time, car journey share and reuse sensor data sets by describing sensors duration) correspondence to physical time. and data in a light-weight syntactic model. PI sup- The most common mechanical representation for ports classifying raw sensor data into high-level con- physical time is the ISO 8601 standard [95], which text through user-specified rules. SDDL and PI focus is based on the Gregorian calendar, and provides a on data, while more standard solutions like SensorML lexical format for modelling dates, times, durations, and SSN aim to standardise interfaces for services and time intervals, and time zones. For example, 2012- description languages for sensors and their processes 05-10T09:32BST represents the time of 9:32am on to enable syntactic interoperation [48]. Sensor ontolo- the 10th of May, 2012 in British Summer Time. A gies are moving towards a deeper knowledge model subset of the lexical formats defined by this standard like OntoSensor and SWAMO [213] to express and au- are adopted by XML schema, which RDF adopts as a tomatically create a composition of processes of sen- means of typing data literals. sors [48]. To do this, the current sensor ontologies shall Given any two temporal features, there exists a rela- work towards enabling a variety of uncertainty rea- tionship between them that we may also wish to model. soning mechanisms (e.g., fuzzy and probabilistic rea- Instants are related to intervals by the notion of con- soning) and more advanced technologies like mixtures tainment, while Allen’s temporal calculus [4] defines of DL, structural similarity and information retrieval the following seven relationships between time inter- techniques [48]. vals – during(t1, t2): time interval t1 is fully contained 5.2. Modelling and Querying Context within t2; – starts(t1, t2): time interval t1 shares the same be- The nature of pervasive computing is such that ginning as t2, but ends before t2 ends; the applications encompassed by this broad terms are – finishes(t1, t2): time interval t1 shares the same vastly heterogeneous in nature and broad in their data end as t2, but begins after t2 begins; requirements and scope. Hence, in an open environ- – before(t1, t2): time interval t1 is before interval ment it is impossible to define, without a priori knowl- t2, and they do not overlap in any way; edge of the applications that will use it, a ‘complete’ – overlap(t1, t2): interval t1 starts before t2, and model of content. However we may straightforwardly they overlap; observe that a large number of applications exhibit – meets(t1, t2): interval t1 is before interval t2, but overlapping data requirements, the most common of there is no interval between them, i.e., t1 ends which are the need to represent time, location, actors, where t2 starts; and resources. A vocabulary for time supports the tem- – equal(t1, t2): t1 and t2 are the same interval. poral localisation of entities; in supporting spatial lo- and their inverses (contains, startedBy, finishedBy, af- calisation, location has multiple representations, each ter, overlappedBy, and metBy) for a total of thirteen suited to particular classes of application; people and relationships (equals being its own inverse). agents are central actors in pervasive applications; and, The W3C OWL-Time ontology [141], which pro- the notion of a resource encompasses myriad things, vides a vocabulary for Allen’s thirteen temporal rela- both physical and virtual with which a user or applications, uses XML Schema’s dateTime formats, but tion may interact. These neatly correspond to the no- also provides its own component based on Date- tions of when, where, who, and what introduced in sec- TimeDescription that can express additional in- tion 2.3. formation (e.g., ‘day of week’ and ‘day of year’) to 18 Ye et alia / Semantically-enriched pervasive sensor-driven systems make mechanical extraction of and reasoning on this tioning between entities based on compass directions information easier. However, we note that as a tem- and distance. The LOC8 framework uses this model poral relationship is formed by each pair of temporal to support conversion of positioning data between co- entities, it is impractical to manually specify or con- ordinates, symbolic locations and relative positions. It cretely realise these in any data model of notable size. also provides an API for querying for the position of This indicates a need for special consideration when entities, returning a list of entities within a spatial re- evaluating these temporal predicates as part of a query. gion, and generating paths between two points in the Of the ontology modelled systems described in the model [179]. previous section, SOUPA’s temporal predicates are 5.2.3. Who—Person/Agent based on DAML-Time (OWL-Time’s predecessor), This context type broadly relates to the actors in a while Ontonym adopts OWL-Time directly in its mod- pervasive system, which are typically people or agents elling of temporal concepts. Temporal concepts may whose reputations are manifest by software. The data be straightforwardly mapped to the top level ontolo- representation requirements associated with these enti- gies discussed. Although it is possible to represent all ties are usually application (or a class of applications) the temporal relations, only a subset of the relations specific, and focus on the role played, or interactions map to the properties of the underlying model (e.g., expected of an entity in a scenario. during or overlap) upon which standard inference is The CONON person ontology is tightly-coupled to performed. its application offerings and provides a small vocab- 5.2.2. Where—Location ulary for describing a person’s name, age, and situa- Location information provides a means of invoking tion. SOUPA builds upon the FOAF [65] to support the application behaviour based on the real world position- expression of human profile information (name, age, ing of users and artefacts. Location information has a contact details), and the MoGATU BDI ontology [147] number of possible representations, ranging from ab- to support the description of agent state – their beliefs, solute, to relative, to symbolic, all of which can be re- desires, and intensions – the detail of which is an appli- lated, and with specific forms more appropriate than cation specific concern. Ontonym’s person vocabulary 3 4 others for any given application [60]. Location is often is based on a subset of terms from vCard , W3C PIM regarded as the most important type of context, and lo- and FOAF, and provides support for modelling date of cation models and query frameworks had been exten- birth, gender, language, and contact profiles, postal and sively researched, before the emergence of the Seman- email addresses, telephone and fax numbers, and web tic Web (e.g., [89,87,100,155]). presence. Ontonym provides a component based name The CONON location ontologies exist primarily to model that supports the semantic modelling of name differentiate between indoor and outdoor spaces, while terms (e.g., ProfessionalTitle, GivenName, PatronymicName ASC’s location ontologies have limited scope, support- and ), and borrows from Davies ing the representation of spatial positions in either the and Vitiello’s relationship vocabulary for FOAF [71] WGS84 and Gauss-Kreuger coordinate systems, and to define relations between people covering genetic, working, romantic, residential, and friendship connec- representing the distances between positions. tions. More comprehensive is SOUPA’s location ontology, which is formed through the conjunction of two ex- 5.2.4. What—Resource isting location vocabularies, OpenCyc [111] and Re- Given that the term resource is itself generic, the gion Connection Calculus (RCC) [23,46]. The Open- lack of an all encompassing vocabulary for describing Cyc vocabulary supports the symbolic representation resources is not surprising. A resource may refer to of spaces, while RCC provides a set of spatial rela- anything not otherwise explicitly modelled in a partic- tions (e.g., overlap, disconnection, tangental part) that ular pervasive system, for example, a device, an image, may hold between two regions—essentially the loca- a document, a physical object without computational tion equivalent of Allen’s temporal relations discussed power, or a virtual artefact. While, for the most part, above. Ontonym [177] adopts a simpler spatial re- the properties of a resource one might wish to model lation model, supporting only containment, overlap, are application specific in nature, resource descriptions and adjacency. It supports multiple coordinate systems (based on the coordinate translation scheme of Jiang 3vCard ontology: http://www.w3.org/2006/vcard et al. [100]) and incorporates a notion of relative posi- 4W3C PIM: http://www.w3.org/2000/10/swap/pim/ Ye et alia / Semantically-enriched pervasive sensor-driven systems 19 might draw from multiple ontologies. For example, ing interoperability problems and providing the means FOAF provides some general terms for describing im- for high-level event interpretations based on ontol- ages, documents, and projects, the Dublin Core Meta- ogy reasoners. This section reviews existing domain- data Initiative vocabulary5 provides terms for describ- independent ontologies for event modelling, present- ing aspects of resources such as their creators, repre- ing the different design patterns and expressive capa- sentation format, and licensing, and the GoodRelations bilities they provide. In addition, research efforts to- ontology [85] provides an e-commerce vocabulary for wards the development of semantic complex event pro- describing products and services. cessing (SCEP) frameworks are described that enrich 5.2.5. Analysis the results of traditional CEP engines with ontology- This section has evaluated the way in which the based background knowledge and reasoning. most common elements of primary context in pervasive computing are represented. While one can feasi- 5.3.1. SOUPA bly conceive of the widespread adoption of a single, Events in SOUPA are modelled as instances of the shared temporal model and location model upon which soupa:Event class that represents a set of all events event descriptions are based, the specification of the in the domain. However, this class is time and space actors and resources in a pervasive environment will agnostic in the sense that it does not provide proper- remain highly application specific. Where the need to ties for directly associating events with temporal and model particular features of such entities occurs sep- spatial information. arately (i.e., using separate vocabularies), the promise Spatiotemporal information in SOUPA can be mod- of Linked Data [19] plays a critical role in support- elled in terms of the soupa:SpatialTemporal- ing the integration of entities with diverse descriptions Thing class that is defined as the intersection of the across applications and environments. soupa:TemporalThing and soupa:Spatial- Thing classes. Particularly for events, the soupa:- 5.3. Event-centric modelling and reasoning SpatialTemporalEvent class is defined as the intersection of the soupa:Event and soupa:Spa- The efficient representation and processing of events tialTemporalThing. is an important and challenging task in pervasive environments. In most cases, only a small number or a 5.3.2. Ontonym combination of raw primitive events is of real interest. The event ontology of Ontonym provides a means Such low-level events are produced directly by the sen- of describing activities of interest within a system. The sors or after low-level data processing. However, high- ontonym:Event class is defined as the union of the level, real-world complex events, such as situations ontonym:InstantEvent and ontonym:Inte- and human activities in Figure 1, cannot be recognised rvalEvent classes that are used to model events or processed, due to the absence of an expressive do- that occur instantaneously or over a time period, remain knowledge that would enable the further corre- spectively. Spatial information on events can be ex- lation of events and multi-modal information beyond pressed through the ontonym:SpatioInstant- predefined patterns and attributes. Moreover, in het- Event and ontonym:SpatioIntervalEvent erogeneous environments with a large volume of sen- classes. Any temporal information in Ontonym is ex- sors where data is gathered from heterogeneous systems with potentially different terminologies, the rep- pressed using the ontology constructs of the OWL- resentation, integration and correlation of event knowl- Time ontology. Spaces are represented using a combi- edge raise new challenges. nation of ontonym:SymbolicRepresentation, In order to overcome the above limitations, research ontonym:GeometricRegion and ontonym:- efforts have focused on the definition of ontology- RelativeLocation ontology classes. based event models. Ontologies are used in this con- A role played by a person, device, resource, or ob- text as common vocabularies for representing knowl- ject in an event is represented as an instance of the edge relevant to events and their relationships, solv- ontonym:Role class. The ontonym:contains- Role property identifies the types of roles that can 5DCMI terms: http://dublincore.org/documents/ be taken in an event and the ontonym:playsRole dcmi-terms/ property is used to associated an entity with an event. 20 Ye et alia / Semantically-enriched pervasive sensor-driven systems the Simple Event Model (SEM) Core Classes Literal sem:hasTimeStamp sem:Core

sem:hasPlace

sem:has sem:Event sem:hasActor sem:Actor sem:Place sem:Time SubEvent

sem:hasTime

sem:eventType sem:actorType sem:placeType sem:timeType sem:hasTime

sem:EventType sem:ActorType sem:PlaceType sem:TimeType T ype System (Foreign)

Literal sem:hasTimeStamp sem:Type sem:hasSubType

sem:hasTimeStamp sem:RoleType Constraints Property sem: sem:Constraint sem:Authority sem:roleType accordingTo sem:View

sem:Role sem:Temporary Willem Robert van Hage Véronique Malaisé Vrije Universiteit Amsterdam

Fig. 6. The classes of the Simple Event Model ontology [203]. Arrows with open arrow heads symbolise rdfs:subClassOf properties and regular arrows visualise rdfs:domain and rdfs:range restrictions

5.3.3. The Event Ontology 5.3.4. Simple Event Model The Event Ontology (EO)6 is a simple upper-level The Simple Event Model (SEM) [203] is an effort ontology for modelling events, defining three classes to define an ontology model for events without strong (eo:Event, eo:Product and eo:Factor) and semantic constraints. This decision is justified by the eighteen properties. An event may have sub-events open nature of the Web and the need to model different (eo:sub_event), actively participating agents (eo:- (even conflicting) views of the same event. The lack, however, of strong semantic constraints, such as func- agent), passive factors (eo:factor), products tional properties, disjoint classes and cardinality re- (eo:product, i.e. everything produced by an event), strictions, hampers the ability to automatically validate and a location in space (eo:place) and time (eo:- 7 and resolve model inconsistencies using formal infer- time) by reusing the WGS84 Geo Positioning and ence mechanisms. Therefore, SEM is characterised by OWL-Time ontologies, respectively. a tradeoff between model reusability and automated The Event Ontology has been mainly used for defin- reasoning and validation capabilities. Figure 6 presents ing and interlinking events in the LOD cloud [173] and the relationships among the classes of the SEM ontol- for annotating and extracting multimedia events, such ogy. as recordings and compositions, being part of the Mu- SEM defines four core classes: sem:Event (for sic Ontology8. For example, an extension of the Music modelling events), sem:Actor (who or what partici- Ontology has been used in [175] in order to develop an pated in an event), sem:Place and sem:Time (cap- ontology-based music recommendation system based turing where and when the event takes place). Each on user’s emotions/moods that combines user profiles core class is associated with the sem:Type class that and event (situation) reasoning. is used to aggregate implementations of type systems from other ontologies, such as the Getty Thesaurus of Geographic Names9. Therefore, the instance types in SEM can be either instances or classes from for- 6http://motools.sourceforge.net/event/ event.html 7http://www.w3.org/2003/01/geo/wgs84_pos 9http://www.getty.edu/research/tools/ 8http://www.musicontology.com vocabularies/tgn/index.html Ye et alia / Semantically-enriched pervasive sensor-driven systems 21 eign vocabularies, targeting at the reusability of exist- Participation Pattern. This pattern is used to model ing type vocabularies, regardless of how the types are the participation of objects (dul:Object) in events implemented. (dul:Event). The participation is expressed by an There are two main property types in SEM: sem:- instance of the F:EventParticipationSituat- EventProperty and sem:type. The former is ion class that satisfies an instance of the F:Eve- used to correlate instances of the sem:Event class ntParticipationDescription class. The sit- with other instances of arbitrary classes and the lat- uation includes the event and the participating objects, ter correlates instances of the sem:Core class with while the description classifies the event (dul:EventT- ype dul:Role instances of the sem:Type class. There are also ) and the participants ( ). two aggregation relations. The sem:hasSubEvent Mereology Pattern. Compositions of events are ex- can be used to define specialisation of events. Sim- pressed using the mereology pattern (Figure 7). The ilarly, the sem:hasSubType relates instances of composite event and the component events are clas- the sem:Type class. Finally, there are seven time- sified by the F:Composite and F:Component related properties: sem:hasTimeStamp (for sin- classes, respectively, and they are all referenced by a gle time values), sem:hasBeginTimeStamp and situation instance (F:EventCompositionSitu- sem:hasEndTimeStamp (for time intervals), sem: ation). The situation instance satisfies a description hasEarliestBeginTimeStamp, sem:hasLate instance (F:CompositionDescription) that clas- stBeginTimeStamp, sem:hasEarliestEndTi sifies the composite and component events. Further- Correlate more, events may be temporally related to each other meStamp and sem:hasLatestEndTimeStamp and such relations can be expressed using the DOLCE (for uncertain time intervals). vocabulary. D5.2.1 - V1.0 Property constraints in SEM can be defined ei- defines ther as a reification of the property or by turning the EventCorrelationDescription property into n-ary relation. The classes sem:Role, Concept sem:Temporary and sem:View are three types of Description EventType Role SEM constraints for defining the role of a class in- satisfies stance (e.g., sem:Actor) in an event, the temporal Correlate Justification boundary within which a property holds and the dif- classifies classifies Situation ferent points of view, respectively. More expressive re- Event Description lationships among events, such as causality [168], or isEventIncludedIn active/passive actor participation semantics cannot be EventCorrelationSituation isObjectIncludedIn modelled in SEM. Fig. 8. The correlation pattern in Event-Model-F (source: [169]) SEM defines also mappings of the model on other Figure 7: Correlation Pattern ontologies, e.g., LODE [173], based on the SKOS vo- Causality Pattern. In Event-Model-F, causes and ef- cabulary. Furthermore, the developersships. provide Thus, aPro- using a single causality relationship might not be precise enough. Consequently, fects are events and the causality pattern is used to ex- log API for SEM in order to ease theher procedure logic for of pop- causality allows for explicitly modeling the different kinds of causal relationships ulating the ontology with instances. press relationships between events that play the roles found in literature. Thisof can causes be supported and effects. with The ourF:Cause Event-Model-Fand F:Effect by specializing the causality 5.3.5. Event-Model-F pattern to specific kindsclasses of causal are usedrelationships. to classify events as causes and ef- Event-Model-F [168] defines an expressive model fects, respectively, and the F:Justification class for capturing and representing occurrences in the real is used to classify a dul:Description instance world. It is based on DUL, following3.4.4 the descriptions Correlation Patternunder the justification of some theory. and situations ontology design pattern (DnS) [73] for Correlation Pattern. Two or more events are corre- modelling aspects of events, suchA as set object of events partici- is calledlated correlated if they have if they a common have a cause common and cause.there is However, no there exists no pation, mereological, causal, and correlativecausal relationship relation- betweencause-effect the two relationship events [ among57]. The them common (causality). cause The may originate from a ships, and different interpretationssingle of the orsame a chain event of multiplecorrelation preceding pattern cause-effect defines the relationships. role F:Correlate Correlation also differs from (by reifying events in order to describeco-occurrence n-ary relations). where twothat orclassifies more events justcorrelated (randomly) events. happen Similarly at the to same time and do not The Event-Model-F ontology introduceshave a six common ontology cause. the causality pattern, the F:Justification role design patterns that are described briefly in the rest of classifies a dul:Description instance that this section. Correlation is not of metaphysicalexplains the correlation. interest as it is a property that can be derived from causality, i.e., the common cause. In our ontology, we model correlation explicitly as in many cases the (correlating) effects of some common cause may be known, while the cause itself is not. The correlation pattern depicted in Figure7 defines the role F:Correlate to classify the events that are correlated. The Justification role classifies some Description, which explains the correlation in terms of a (mathematical) law or some theory.

3.4.5 Documentation Pattern

Documentary evidence for an event may be given by arbitrary objects, e.g., some sensor data or media data, or by other events. Formally, this relation is expressed by the documentation pattern depicted in Figure8. It defines the concept F:DocumentedEvent that classifies the documented event and the concept F:Documenter that classifies the documentary evidence for that event. This evidence can be expressed by any specialization of an Object, e.g., a digital photo taken with a cell phone during an incident, or a specialization of Event. Thus, the documentation pattern enables the inclusion of evidences described in ontologies other than F. For example, digital media data like images and videos can be classified as Documenter and precisely described using, e.g., the Core Ontology for Multimedia [2, 60]. Thus, the documena- tion pattern connects the representation of the real-world events with the media assets that were taken during these events. The objects are documented via the events in which they participate (see participation pattern in Section 3.4.1).

Page 28 D5.2.1 - V1.0

3.4.2 Mereology Pattern

Events are commonly considered at different abstraction levels depending on the view and the knowledge of a spectator. For instance, the event of a flooded cellar may be considered as such or as part of the larger event of a flooding in which many other (smaller) in- cidents occur. The mereology pattern shown in Figure5 enables expressing such mereological relations as composition of events. The composite event is the “whole” and the component events are its “parts”. Formally, a F:EventCompositionSituation includes one instance of an event that is classified by the concept F:Composite and many events classified as its F:Component(s). Accordingly, an EventCompositionSituation satisfies a F:CompositionDescription that defines the concepts Composite and22 Component for classifyingYe et alia / the Semantically-enriched composite event pervasive and sensor-driven its component systems events.

defines EventCompositionDescription

defines Concept Parameter

Description EventType EventCompositionConstraint isParameterFor

Composite Component TemporalConstraint classifies classifies parametrizes SpatioTemporalConstraint satisfies TimeInterval parametrizes Event hasParticipant SpatioTemporalRegion isEvent IncludedIn Object SpatialConstraint hasQuality hasQuality parametrizes Situation hasRegion Quality SpaceRegion isTime isSpaceTime isSpace EventCompositionSituation IncludedIn IncludedIn IncludedIn

Fig. 7. TheFigure mereology 5: pattern Mereology in Event-Model-F Pattern (source: [169]) Documentation Pattern. This pattern is used to pro- and seven properties. In contrast to Event-Model-F, Eventsvide documentary that play the evidencesComponent for events.role The may docu- be furtherLODE qualified focuses byon representingtemporal, spatial, stable characteristics and spatio- temporalmented event constraints. is classified As by events the class areF:Docume- formally definedof as events, entities excluding that exist parthood in time and causal and not relations in space that (cf.ntedEvent Section and3.1 the), constraintsdocumentary evidence including is classi- spatial restrictionsbelong to the are interpretative expressed dimension through of the event objects mod- fied by the class F:Documenter. The evidence may participating in the component event. For instance,elling. a Component event may be required to be a specialisation of the Object class or another The lode:Event class is defined as subclass occurevent. within a certain time-interval, e.g., the secondof the weekE2_Temporal_Entity of June 2009. Dependingclass of CIDOC on its objects,Interpretation a Component Pattern. The perceptionevent may of events also happenheav- inand a equivalentcertain spatial to the eo:Event region. Forand example,dul:Event the floodingily depends of ona town the context should and be the composed viewpoint of the events thatclasses. have In objects this way, associated an event in to LODE it, which represents have someobserver certain and the range explicit of modelling longitude of and such latitude. contex- Finally,something events that happened and the over objects a limited bond extent to init time may without specifying a change of state or attempting to betual qualified views is crucial. by a spatio-temporal In Event-Model-F,quality such different like the progress of a flood that extents over time and viewpoints can be expressed by instantiating accord- distinguish events from processes or states. space,ingly the starting aforementioned with a highdesigned water patterns level and located bind- in someIn LODE, area anof event a river can and be associated extending with spatially at most overing them time with into the other interpretation areas. pattern. Any such The pattern constraintsone are interval formally of time expressed through the bylode:atTime one or multipleprop- instancesdefines the ofF:Interpretant the F:EventCompositionConstraintclass that classifies erty that is. defined They define as subproperty qualifying of the values CIDOC’s for temporal,the event under spatial, interpretation. and spatio-temporal Each interpretation qualities fur- ofP4.has_time-span a component eventand asdul:isObservableAt shown in Figure5. ther classifies the relevant situations, that is, the par- properties. It is also equivalent to eo:time and the Thus,ticipation, with mereology, the composition causality, correlation, pattern, events and docu- may betime arbitrarily intervals aretemporally defined using related the TemporalEn- to each other, i.e.,mentation they might pattern be instantiations. disjoint, overlapping, or otherwisetity ordered.class of the In OWL-Time order to express ontology. such relative temporal5.3.6. Linking relations Open Descriptions between events, of Events one can facilitateThe the providedlode:inSpace meansproperty of DOLCE relates such an as event the formalizationThe Linking ofOpen Allen’s Descriptions Time of Calculus Events (LODE)8. to some spatial boundary (at most one), i.e. a region of space, and it is defined as a subproperty 8 ontology [173] provides a model for representing and http://wiki.loa-cnr.it/index.php/LoaWiki:Ontologiespublishing events that have happened in the past as of dul:hasRegion, equivalent to eo:place and Linked Data [19]. The model has been designed in super-property of CIDOC’s P7.took_place-at such a way, so as to allow the modelling of four ba- property. Events can be also related to abstract places sic aspects of events: what, where, when, and who.Page 26through the lode:atPlace property that is sub- The LODE vocabulary is aligned with other eventproperty of dul:hasLocation. related vocabularies and ontologies, such as DUL, The association of events with objects and agents Event Ontology (EO) and CIDOC [61], and consists is performed thought the lode:involved prop- of one class for representing events (lode:Event) erty (equivalent to DUL’s hasParticipant) and 4 Kia Teymourian, Malte Rohde, Ahmad Hasan, and Adrian Paschke

knowledge updates is not very high as the rate of the main event stream, e.g., frequency Ye et alia / Semantically-enrichedof happening pervasive of music sensor-driven concerts systems compare to changes in a music band. 23 its subproperty lode:involvedAgent (equivalent to DUL’s involvesAgent and super-property of eo:agent), respectively. 5.3.7. Semantic Complex Event Processing State-of-the-art Complex Event Processing (CEP) frameworks [163] are able to efficiently recognised complex events based on predefined event-patterns, event-hierarchies or other event relationships (e.g., temporal). CEP engines are capable of binding to input streams of real-time structured events that origi- Fig. 1. High LevelFig. 9. Architecture An abstract of SCEP Semantic-Enabled architecture (source: Complex [143]) Event Processing nate either directly by sensors or after low-level feature processing. The data of the event streams are checked in terms of atomic events and the alerts that should be for specific values or correlations with data from other triggered upon detection. These events are transformed streams in order to identify and extract useful patterns into CEP streams and queries for the configuration of that are described using rules. The key feature is4 the Experimentsthe CEP and engine. Demonstration support of temporal relationships (e.g., Allen’s tem- Teymourian and Paschke4 [195] present a framework poral interval relationships) and aggregation operatorsFor our experiments,for semantic we use rule-based Prova as reaction complex rule events language processing. formalization and as a rule-based execution which can be used as event processing engine. Prova uses reactive that enable the identification of complex correlations 5 The ontology based representation6 of event data is per- among the generated events. messaging , reaction groups and guards for complex event processing. Multiple mes- sages can beformed revived using by a revMult(XID, map of the plain Protocol, attribute-value Destination, Performative, pairs to Payload) Semantic Complex Event Processing (SCEP) is; XID, an a conversationa set of RDF id of the triples message; (RDF Protocol, graph) messagethat can passing be used protocol; as Destina- effort to improve the results of CEP by incorporatingtion, an endpoint;event Performative, instances. In message this way, type; events Payload, are the represented content of in message. Prova ontologies and rules into the process of complex eventimplements aterms new inference of URIs extension and they called can be literal further guards. interlinked During the with unification only detection [128]. An abstract CEP architecture isif pre- a guard conditionother ontology-based evaluates to true, the domain target ruleknowledge. will proceed More with com- further evaluation. 7 sented in Figure 9. Ontologies are used in thisWe con- implementedplex the eventssparql_select can bebuilt-in retrievedtoby runexecuting SPARQL queries SPARQL from Prova which can start a SPARQL query from inside Prova on an RDF file or a SPARQL endpoint. text as common vocabularies for representing knowl- queries that match complex graph patterns. The reac- This buit-in can use results which come from the SPARQL query and use them inside edge relevant to events, such as event patterns, par- tion rule language of Prova11 is used in order to im- ticipants and their relationships, solving interoperabil-Prova. It also provides the possibility to replace variables in SPARQL string which are starting withplement $ with variables. temporal reasoning operators and to perform ity problems. Moreover, low-level events can be se-In our experimentsreasoning we on use the the ontology-based Prova rule engine. knowledge. We use the sparql_select built-in: mantically associated with high-level domain conceptsthe rule engineETALIS first sends [6] the isembedded a CEP SPARQLengine implemented query to triple in store, Pro- gets the results of background ontological knowledge, improvingback the and thenlog waits that for supports incoming the event definition stream to and process. execution It processes of EP- the sequence of quality of event and activity recognition using contex- 4 Prova, ISOSPARQL Prolog syntax queries, with extensions as wellhttp://prova.ws as the use of background, July 2011 tual information. Usually, the semantic association5 Prova of Reactiveknowledge Messaging in ahttp://www.prova.ws/confluence/display/RM/ form of RDF ontologies [5]. EP- the events with background knowledge is further en-Reactive+messagingSPARQL is a, language July 2011 for event processing and stream 6 hanced with rules so as to overcome expressive lim-Event Processingreasoning Using that Reaction is usedGroups in orderhttp://www.prova.ws/confluence/ to detect events within a itations of the underlying ontology languages, as de-display/EP/Event+processing+using+reaction+groups, July 2011 7 stream of RDF triples (low-level events). EP-SPARQL scribed in section 3.4. In this way, the native capa-Source codes for Semantic Web extensions in Prova 3 can be found in https: //mandarax.svn.sourceforge.net/svnroot/mandarax/prova3/extends SPARQL with a set of binary operators (SEQ, bilities of CEP engines for real-time event processingprova-compact/branches/prova3-sw/EQUALS, OPTIONALSEQ, and, JulyEQUALSOPTIONAL 2011 ) can be integrated into ontology-based pervasive frame- used to combine graph patterns, as well as, with a set works for time-aware and high-level interpretation of of functions (e.g. getDURATION, getSTARTTIME) sensory data. 10 for time-based filtering expressions. The queries are In [192], a commercial CEP engine (Coral8 ) is compiled into event-driven backward-chaining rules combined with ontology-based reasoning. The plat- that can be mixed with other background knowledge. form aims at the semantic configuration of the CEP engine using domain ontologies about events, sensors, 5.3.8. Analysis environments, etc. The event ontology extends SSN Events capture the dynamic aspects of a domain and it is used as a repository of complex event defini- and their efficient representation, processing and anal- tions that users define. Complex events are described ysis are considered key requirements in pervasive and sensor-driven environments. The motivation behind 10http://www.aleri.com/products/aleri-cep/ coral8-engine 11http://http://prova.ws/ 24 Ye et alia / Semantically-enriched pervasive sensor-driven systems

Table 4 Comparison of the event ontologies presented in section 5.3

Ontologies Time Location Participation Mereology Causality Correlation Interpretation Axiomatisation Modularity √ √ √ √ SOUPA –––– moderate √ √ √ Event Ontology limited limited –– limited – √ √ √ √ Ontonym –––– moderate √ √ √ √ SEM limited –– limited – √ √ √ √ √ √ √ √ Event-Model-F rich √ √ √ LODE –––– limited – the development and use of ontology-based event mod- requisites towards understanding a user’s state, be- els is to provide formal and explicit vocabularies able haviour and surroundings. Espousing the ontology- to semantically represent and correlate common as- based modelling paradigm, the low-level context in- pects of events (e.g., places, people, objects) amenable formation directly acquired from sensors is translated to reasoning, and thus to high-level interpretation. to respective ontological class and property assertions. This section brieﬂy presented existing event ontolo- Hence, typical ontology reasoning tasks can be used gies, focusing on the ontology constructs and design to check the consistency of the aggregated set of con- patterns they provide. The representation of common textual assertions and, more importantly, to derive aspects of events, such as time, location and participa- more complex context abstractions (e.g., recognise a tion, is supported by all ontologies. However, the rep- user’s activity based on her current location and ob- resentation of more complex event relationships, such jects used) that would otherwise remain implicit. An as mereological or causal relationships, are fully sup- abstract ontology-based framework for modelling and ported only by the Event-Model-F ontology that pro- reasoning about high-level context is depicted in Fig- vides a rich axiomatisation of ontology design patterns ure 10. (section 5.3.5). SOUPA and Ontonym follow a mod- In addition to adopting plain ontology-based solu- ular design with a moderate axiomatisation compared tions though, a number of approaches have explored to the Event Ontology, LODE and SEM ontologies. reasoning frameworks that combine ontologies and Among them, only SEM is capable of capturing dif- rules [17,15] in order to cope with OWL’s restricted ferent interpretations of the same event. Table 4 sum- relational expressivity (see Section 3.4). Based on the marises the strong and weak points of each ontology. level of interaction that different combination frame- In addition, this section introduced the notion of se- works afford, these approaches can be discriminated mantic complex event processing (SCEP), an effort to- into three categories: maximising the interaction be- wards the combination of Semantic Web technologies tween the ontology- and rule-based inferences; adher- and traditional complex event processing frameworks. ing to a tight integration; and adopting a much looser The motivation behind this combination is to enrich the notion of coupling. In the following, representative pattern-based complex event detection capabilities of frameworks for each of the three paradigms are dis- CEP with reasoning over ontology-based event mod- cussed. els and background knowledge. In this way, complex 5.4.1. Ontology-based frameworks events can be detected and interpreted based on their The CoBrA and the Semantic Smart Home (SSH) hierarchical or temporal relationships to other events, frameworks described in Section 4 are both examples as well as, on their correlations with other relevant con- of plain ontology-based approaches to reasoning about cepts of the domain, e.g., a relationship between a car context. In particular, in CoBrA, through a set of on- accident event and the location of emergency vehicles. tologies, useful inferences can be drawn regarding a person’s location and activity context. For instance, 5.4. Context Reasoning the detection of Alice’s mobile phone in a speciﬁc location entails Alice’s presence [34]. Jena is used to The higher-level integration of raw context data manage and reason over the knowledge base, while and comprehension of their meaning are key pre- reasoning is invoked through RDQL that periodically Ontology Class/property assertions Reasoner

Low-level context Pervasive information from sensors Application ABox

TBox Complex context abstractions (e.g. user’s activity)

Ye et alia / Semantically-enriched pervasive sensor-driven systems 25

Class/property Ontology assertions Reasoner

Low-level context Pervasive information from sensors Application ABox

TBox Complex context abstractions (e.g. user’s activity)

Fig. 10. An abstract ontology-based framework for high-level context reasoning from low-level sensory data queries the knowledge base for the presence of cer- dividuals and realisation is used to determine into tain context data; e.g., whether a device has been de- which context concepts a specific situation individ- tected in a specific room, or who is the owner of ual falls (e.g., AuthorisedPersonRinging). Evaluations the device. Once queries return matched results, new with three DL reasoners (Racer, RacerPro and Pel- assertions about the local context are added to the let) demonstrate the feasibility of the proposed frame- knowledge base. In an alternative implementation, an work; yet, the examined scenario is rather simple to object-oriented rule-based reasoning engine has been safely extend conclusions regarding its evaluation in developed using Flora-2 [32]. CoBrA also includes a much broader context-aware applications. Towards a Prolog-based meta-rule reasoning component to en- more effective engineering the domain knowledge, a force privacy policies tailored to user needs and pref- generic methodology for situation modelling has been erences based on respective sets of user-defined rules. proposed in a subsequent work [176]. The methodol- The ontologies defined within the SSH framework ogy is based on the systematic decomposition of situ- capture knowledge related to physical equipment (such ations according to so called of aspects of interest that as sensors and electrical appliances), actions and ac- consider spatial, temporal and acting person criteria, tivities of daily living (such as watching television and and aims to assist system developers in effectively cap- preparing a meal), location spaces (such as kitchen turing situations at different levels of granularity. Such and living room), people and their roles, medical in- modelling allows to effectively avail of the subsump- formation, software components as well as tempo- tion semantics and derive inferences (though more at ral information [40]. Modelling activities of daily liv- coarser levels of abstraction) when not all pieces of the ing (ADLs) through property restrictions in relation to relevant context information are available. equipment, location and other types of constraints, al- A more recent example is the OWL 2 based con- lows the inference of ADLs on the basis of the as- text modelling and reasoning architecture presented in sertional knowledge made available through sensors. [160]. As in the aforementioned initiatives, ontologies For example, an ADL a is inferred to be an instance are used not only to represent activities but also rel- of KitchenADL ≡ ∃hasLocation.Kitchen, evant knowledge that can drive their recognition, in- if there is sensor evidence that asserts hasLoca- cluding locations, objects, and so forth. Unlike the pre- tion(a, kitchen). Further inferences, of higher- vious approaches though, sensors are directly mapped level of detail can be drawn as additional sensor de- to ontological classes and properties and the sensor scriptions are made available. Continuing the exam- data are directly added to the assertional part of the ple, the axioms MakeDrink v KitchenADL u knowledge base. Sensor data are first fed to COSAR ∃hasContainer.Container and Cup v Con- [159], where primitive activities through a combina- tainer, enable to further classify a as an instance tion of statistical and ontological reasoning are de- of the MakeDrink class, once the contact sensor at- rived. These simple activities, in combination with fur- tached to a cup is activated. ther contextual data, once aggregated and processed so In [201], an ontology-based framework for context- as to resolve possible conflicts [1], formulate subse- aware applications in the smart home domain is pre- quently the assertional knowledge, over which OWL sented using as case study a door lock scenario. The 2 DL reasoning can be applied in order to recognise concrete situations (i.e. the contextual information di- more complex activities. A worth noting feature of the rectly acquired by sensors) correspond to OWL in- proposed framework is that by exploiting OWL 2’s 26 Ye et alia / Semantically-enriched pervasive sensor-driven systems support for composition of properties and for qualified the Beliefs, Desires and Intentions (BDIs) agent model cardinality restrictions, the captured knowledge is con- [158]. Provided with enriched contextual knowledge siderably more expressive compared to that afforded that would otherwise remain implicit, BDI agents can by earlier frameworks that use OWL 1 DL. Despite the make improved decisions and provide better services restrictions imposed on the use of the property compo- such as negotiation and consultation. sition constructor that conditions decidability, the au- Despite the use of SWRL rules, reasoning in both thors argue that the use of OWL 2 DL is a satisfy- frameworks remains decidable as the existing rea- ing compromise for effectively reasoning, while avoid- soner implementations employ inherently the DL- ing the technical and semantic complexities confronted safety notions, supporting in practice a form of re- when combining ontologies and rules. stricted SWRL. 5.4.2. Tightly-coupled frameworks 5.4.3. Loosely-coupled frameworks An early example of a context reasoning framework Unlike the previously described frameworks, loosely- that combines ontologies and rules can be found in coupled frameworks keep minimal the interaction be- the Gaia infrastructure. Context information in Gaia is tween ontology- and rule-based reasoning. Rules are represented as first-order predicates, with the name of applied to the consequences derived by means of on- a predicate indicating the type of context described. tology reasoning, and affect little the knowledge base. Higher-level contexts can be deduced based on a set In SOCAM for example, ontology-based reasoning is of predefined rules that are reevaluated whenever a used to deduce additional knowledge from the context change occurs [154]. For reasoning in first-order logic, data that are directly acquired through sensors; e.g., the XSB [166] reasoning engine is used. Quantification based on the transitive semantics of the locatedIn of variables is performed over the specific domain of relation, the reasoner can infer that a person is located values, which is finite, ensuring in this way that evalu- inside the house, provided that she is located in the ations of expressions will always terminate. bedroom contained in it. Once implicit knowledge is More contemporary examples include the context- made explicit, first order logic rules are invoked to de- aware systems [222,224] that investigate the use of rive higher-level contexts such as sleeping, cooking SWRL rules. In particular, in [222], OWL and SWRL and watching television [211]. Both reasoning mod- are used to capture and reason over contextual infor- ules are implemented using Jena. A similar rationale mation about museum visitors, including their prefer- has been adopted in the Semantic Space framework ences and surroundings, in order to provide context- [210]. RDQL queries over the context knowledge base aware recommendation services. Racer and Jess com- allow to examine desired contexts so that relevant sets prise the system’s reasoning module, allowing for con- of first order rules can be invoked to derive higher- sistency checking and taxonomic classification (sub- level contexts. The system uses Jena to implement the sumption checking), and the processing of SWRL forward chaining rules and as the inferred contexts are rules and queries respectively. not stored into the knowledge base, it avoids the need Zhang et al. explore the use of SWRL rules in a Se- to handle possible conflicting conclusions. mantic Web-enabled framework for self-management Ontologies and rules have been also coupled in the in pervasive computing [224]. More specifically, a set home-based health monitoring and alarm management of Self-Management Pervasive Service (SeMaPS) on- system proposed by Paganelli and Giuli [140]. The tologies is used to capture salient notions about per- context model consists of four ontologies that cap- sons (including their habits and preferences), loca- ture knowledge about patients (e.g., heart rate and tions, software agents, devices, malfunctions and re- body temperature), home environmental parameters covery solutions, and quality of service parameters and (e.g. humidity), alarm management (including policies dynamic context aspects among others; SWRL rules and contact person information), and social context are used in parallel to capture those parts of com- (e.g., relatives) respectively. Context reasoning is trig- plex contextual knowledge that cannot be expressed gered whenever a change in the context knowledge in OWL. Through the Protege-OWL/SWRL APIs, the based occurs. Ontology-based reasoning is employed RacerPro and Jess reasoning engines are used to de- to determine the complex context class to which a spe- rive context abstractions of higher-level. The infer- cific instance belongs (realisation) and to check the ences drawn are subsequently fed to software agents consistency of the knowledge base once new conclu- that implement BDI reasoning, i.e. reasoning based on sions are added. User-defined rules are employed to Ye et alia / Semantically-enriched pervasive sensor-driven systems 27 trigger alarms and select notification policies based ployed for rules. Such semantics allow the rules to in- on the available biomedical and environmental con- trospect the knowledge base and derive conclusions text data. For example, a rule may assert an alarm ac- based on the absence of information. This induces a tivation when the heart rate frequency is less than 40 non-monotonic behaviour where new inferences may beats/minute and systolic blood pressure is higher than invalidate previously derived conclusions. A represen- 160mm/Hg. Similarly, policy rules are used to assert tative example is given in [160], where a three-room policy-related facts upon detection of alarm activation smart home, equipped with four sensors, one in each facts. Jena has been used to manage and reason over room monitoring the presence of people, and one in the application ontologies and rules. the front door monitoring the entrance of people in the The context-aware access control framework pro- house, is considered. The knowledge base includes the posed in [197] constitutes another example. The frame- following definitions: work follows a hybrid architecture, combining a DL reasoner (Pellet) and a production rule engine (Jess) (1) Room u ¬∃hasOccupant v EmptyRoom in order to apply more expressive context reasoning, (2) EmptyRoom ≡ Room u ¬OccupiedRoom such as property path relationships. For instance, in (3) Room(x) ∧ EmptyHome(y) ∧ a meeting situation there might be the need to model isInside(x,y) → EmptyRoom(x) that if the owner of a requested resource is located in (4) ¬EmptyRoom(x) → OccupiedRoom(x) a certain place and the resource requestor is located in the same place, then the two persons are co-located. In the example, the front door sensor asserts the en- Such relationships are expressed in terms of produc- trance of one person, yet none of room sensors suc- tion (if-then) rules whose head and body match classes ceeds to communicate subsequently that a person is and properties of the ontology. The output of the rules present. Due to the open world semantics of OWL, rule is fed into the DL knowledge base to determine the (4) evaluates to true for all rooms as it cannot be proved value of each attribute given the current context. In this that they are empty. As a result the system ends up in- way, the ontology predicates can be used both in the ferring that there is at least one person present in each body and in the head of rules and thus, the ontology room. knowledge may be modified in iterative ontology and rule-based reasoning steps. 5.4.4. Analysis Though seemingly effective in practice, it is inap- The aforementioned induce two critical observa- propriate to compute the consequences of the ontology tions: component first and then apply the rules to the conse- 1. The combination of ontologies and rules is a key quences [131]. For example, let us assume the follow- prerequisite for effectively meeting the expres- ing rules: sivity requirements when modelling and reason- InsideKitchen(x) ← Cooking(x) ing about context in the pervasive domain; how- InsideKitchen(x) ← HavingDinner(x) ever hybrid reasoning schemes that either only allow for a poor interaction between the two Given the assertion (Cooking t HavingDinner): components or that fail to take into account the Alice, we know that Alice is either cooking or having dinner, but we do not know which one is true. particularities of the co-existence of closed and However, though either way one of the rules derives open world semantics, may easily lead to incor- that Alice is in the kitchen, we cannot deduce it by rect inferences and an overall undesirable be- applying the rules to the consequences of the ontol- haviour. ogy reasoning, as neither |= Cooking(Alice) nor |= 2. The combination of open- and closed-world rea- HavingDinner(Alice) holds. Detecting and prevent- soning is desirable when reasoning about con- ing such unwanted behaviour may be straightforward text in the pervasive domain. The open-world se- for single cases as in the aforementioned example, but mantics are closely related with the capability becomes impractical when dealing with large and com- of reasoning over incomplete knowledge, while plex knowledge bases confronted in real-world appli- closed-world semantics is needed in order to cations. reason over common conjectures about nega- Susceptibility to incorrect inferences is further ag- tive knowledge without having to explicitly state gravated by the close-world semantics usually em- such knowledge. 28 Ye et alia / Semantically-enriched pervasive sensor-driven systems

It is worth noting that ontology-based pervasive ap- Among the ontology-based models in Section 4, plications are not the only research domain where the Gaia and CONON undertake this challenge. Gaia tries need to reason, keeping certain parts of the world open to capture and make sense of the imprecise and con- while closing others, emerges. Similar challenges are flicting data uncertainty inherent in dealing with real- confronted in the semantic understanding of visual world data [154]. An uncertainty model is developed content [54], and in general in applications that require based on a predicate representation of contexts and as- intensional reasoning (e.g., natural language process- sociated confidence values. The predicates’ structure ing). Furthermore, such seamless integration of open and semantics are specified in ontologies that bene- and closed world semantics has been recognised as a fit in checking the predicates’ validity; simplifying the highly challenging yet desirable capability in the Se- definition of context predicates in rules; facilitating in- mantic Web too, resulting in a number of promising teroperation between different systems; and further re- proposals as discussed in Section 3.4. Their practical ducing the possibility of uncertainty when interpret- impact within the pervasive domain remains subject to ing context information. To reason about uncertainty, future investigation. Gaia employs mechanisms such as probabilistic logic, Parallel to this line of research, hybrid architectures fuzzy logic, and Bayesian networks, each of which is that orchestrate formalisms for distinct types of rea- advantageous under different circumstances. For in- soning have emerged. Such investigations begin from stance, Gaia uses Bayesian networks to identify ca- the premise that different aspects of knowledge mod- sual dependencies (represented as edges) between dif- elling (temporal, spatial, etc.) impose different require- ferent events (represented as nodes). The networks are ments that a single, unified formalism is unlikely to trained with real data so as to get more accurate prob- meet. An example is the Situation Awareness by In- ability distributions for their event nodes. ference and Logic (SAIL) architecture proposed in [8], CONON associates metadata to each piece of con- where a theorem prover and a DL reasoner have been text data, describing the quality of each datum. [80] have defined four types of quality parameters: accu- used to reason over streams of time stamped radar data racy, which reflects the estimated error of a measure- and witness reports in order to derive an understanding ment; resolution, which reflects the smallest perceiv- of the overall situation (e.g. a potentially hostile air- able element; certainty, which reflects the probability craft approach), and use linear temporal logic to gen- of the reading being accurate; and freshness, which re- erate appropriate alerts. flects the time a measurement was generated, and its expected lifetime. CONON also introduces a depen- 5.5. Uncertainty dency tag that can be associated with any measurement. This allows relationships to be built between dif- Pervasive computing offers different challenges for ferent contextual information and helps to set down the ontology developers than traditional applications. Data user-defined reasoning rules. in pervasive computing environments may be gener- Both the above approaches use ontologies syntac- ated by untrustworthy or inaccurate sources and so tically as a vocabulary to exchange knowledge base should be taken ‘with a grain of salt’. Because compo- specified in a probabilistic model. Responding to the nents of a pervasive computing environment deal with need of modelling imperfect knowledge in the Seman- the real world, they come with certain caveats: sensors tic Web, much research has been devoted to extend- in the field are inherently inaccurate, since they could ing formalism and reasoning services so as to han- break down; or they could report inaccurately because dle uncertain and/or vague information. Representative they come up against an unusual phenomenon; i.e. one examples including among others fuzzy extensions of for which they have not been designed. DLs [182,181], OWL [180,20] and SWRL [214], and Since these issues must be taken into account when probabilistic extensions such as PR-OWL [52,30] and dealing with pervasive systems, it should be possible BayesOWL [58]; for an extensive overview the reader to describe the concepts of accuracy, uncertainty, and is referred to [183]. Further relevant proposals include provenance with respect to context data and represent the pattern-based approach for representing and rea- them as part of its ontological description. With these soning with fuzzy knowledge [202], and the generic, descriptions in place, particular reasoning mechanisms formalised approach for managing uncertainty pro- on ontologies need to be designed to support efficient posed by Dividino et al. [59]. Few works, however, and precise reasoning on the data. have explored the applicability of such initiatives in the Ye et alia / Semantically-enriched pervasive sensor-driven systems 29 domain of pervasive applications; an example is the grated with probabilistic or fuzzy reasoning. Abduc- approach presented in [45], where fuzzy reasoning is tive reasoning is also worth of further investigation in used to provide personalised mobile services based on that it can help to detect errors (e.g., missing or inac- situation awareness. curate data) which can be used as a feedback to re-tune Missing data is another source of uncertainty when the system. reasoning about context: a miss (or inaccurate) detection of low-level context information may easily lead 5.6. Semantic Service Discovery to irrecoverable failures in the inference of higher-level context abstractions. One possible solution is to model One of the core problems in pervasive computing is the interpretation of perceptual data as inference to the the question of how to support the introduction of new best explanation using abductive reasoning [172,146]. devices into the environment. As these devices may Romero et al. [74,75] investigate this idea in the con- be unaware of the configuration of the environment or text of an ontology-based surveillance application. A the services available they must undergo a discovery set of ontologies are used to capture context at increas- or matchmaking process to best integrate themselves. ing levels of abstractions, including tracking knowl- According to [110], discovery is a process of retrieving edge, scene objects and activities. Once the low-level services that are able to fulfil a task, and matchmak- context acquired from visual sensors is translated into ing is a process of automatically matching semanti- ABox assertions, abductive rules are applied to derive cally annotated services with a semantically annotated missing facts and trigger the derivation of higher-level request. context descriptions. No information is provided what- Gaia uses DAML+OIL to achieve semantic discov- soever about the computational framework used to im- ery as it supports some of the operations required for plement the abductive reasoning and the preference semantic discovery like classification and subsump- criteria used for selecting explanations. Acknowledged tion. It also allows the definition of relations between as a mode of reasoning that is inherent in a plethora of concepts. The use of ontologies and semantic discov- tasks, much research has been devoted to understand- ery replaces scripts and ad hoc configuration files that ing abduction. For a detailed account on the potential were used in Gaia previously. Each entity is associated of abductive reasoning in DLs, the interested reader with a DAML+OIL description that describes its prop- may refer to [63,105]. erties. The ontology server poses logical queries in- A formal model based on defeasible logic is pro- volving subsumption and classification of concepts to posed by Bikakis et al. to support reasoning with im- find appropriate matches. Other entities in the environ- perfect context in ambient computing environments ment may query the ontology server to discover classes [16]. Extending the Multi-Context Systems model of components that meet their requirements. Match- with non-monotonic features, the proposed framework making uses ontologies to determine a set of concepts supports reasoning in cases of missing context knowl- that fulfil the intersection of the requirements of two or edge. Potential inconsistencies are resolved by means more parties, such as a supplier and a consumer using of an argumentation framework that exploits context the matching algorithms described in [199]. The result and preference information that expresses confidence is a set of classes that are semantically compatible to on the contexts considered. The propositional repre- the query class. sentation of context knowledge may not allow a di- Christopoulou et al. [43] uses their GAS ontology rect integration with ontology-based context reasoning for discovery in the event of a component failure. All frameworks; yet possibilities for interesting hybrid ar- components have ontologies describing their interfaces chitectures emerge where contextual assertions can be and available services. If a connection (or synapse) be- selectively translated into equivalent grounded formu- tween two or more components is broken, the ontology las. manager attempts to find an alternative component that offers the same service (i.e., that has the same ontolog- 5.5.1. Analysis ical description) to repair the connection. At present most of ontology-based models in perva- The DAML-based Web Service ontology (DAML- sive computing community are still at the stage of us- S) supplies web service providers with a core set of ing semantic annotations to tag different quality mea- markup language constructs for describing the prop- sures. One of the future directions in dealing with un- erties and capabilities of Web Services [28,29]. It is certainty is to use the ontologies that are tightly inte- used in combination with additional ontologies, which 30 Ye et alia / Semantically-enriched pervasive sensor-driven systems

Service Proﬁle &xsd;#anyURL

serviceClassiﬁcation &xsd;#anyURL &process;#Parameter hasParameter serviceProduct

serviceName &expr;#Condition Proﬁle hasPrecondition textDescription

Process hasResult serviceParameter &process;#Result has_process serviceCategory serviceParameterName ServiceParameter &process;#Input hasInput

Thing sParameter &process;#Output hasOuput ServiceCategory "#hasParameter"/>

DatatypeProperty code SubClass/Property

Fig. 11. Selected classes and properties of the profile in OWL-S [115] organise the concepts appearing in the DAML-S de- tation paradigm of OWL-S for describing inputs, out- scriptions. The use of DAML-S renders the semantics puts, preconditions and effects (IOPEs) by incorporat- of the descriptions machine comprehensible; therefore ing also a number of capabilities for each service. The it enables intelligent agents to discover, invoke and capabilities are considered either as provided, i.e. ca- compose web services automatically. CoOL extends pabilities that are supported by the service, or required, DAML-S with the service context to offer a more for- i.e. capabilities that are needed by the service. [112] mal description of a service’s contextual interoperabil- defines five possible ‘degrees of match’: exact, plug-in, ity. The service context consists of two parts: the con- subsume, intersection and disjoint. These groups indi- text obligation, which specifies the obligations of a ser- cate ‘how good’ a match is. vice in terms of the context of its execution; and the The OWL-S based approaches cannot support finer- context binding, which is used to establish a virtual grained service matching where priorities or weights link from an atomic process of a service to a specific on individual requirements of a service request need aspect of the context [187]. This formal semantic ser- to be taken into account. Also they lack an appro- vice description offers a common understanding of the priate criterion to approximate and rank the avail- relations between services and their associated con- able service with respect to a given request [11]. texts [184]. This facilitates context-awareness and con- To address these issues, [11] extends the request de- textual interoperability during service discovery and scription with a priority value to indicate the rele- execution. vant importance of the individual requirements in a [13] introduces a dedicated Service Discovery Pro- request; e.g., Req v (= 1hasDescription.RD) u tocol (SDP) that enables advertising and discovering (= 1hasP riority.P riorityV alue). The matchmak- services in pervasive environments according to the ing process starts with checking if all mandatory re- semantics of networked services and of sought func- quirements are satisfied by a service. If all of them are tionalities. The approach is based on the Amigo-S met, then the service will be evaluated through approx- language that extends OWL-S [115] (the successor imate matching, where the similarity between this ser- of DAML-S based on OWL) with new classes and vice and the request will be determined depending on properties. Amigo-S enriches the profile-based anno- the semantic deviation of the expected value in request Ye et alia / Semantically-enriched pervasive sensor-driven systems 31 and the available values in the service description for ery individual action in a domain application, CoBrA the same requirement. A similarity score will be as- uses meta-policies that determine behaviour when pol- signed, which will be used to rank the candidate ser- icy rules are not defined. These meta-policies can be vices later. either conservative, in which case the system assumes that all actions are forbidden, or liberal, in which case 5.6.1. Analysis it assumes all actions are permitted. A CoBra proto- Semantic service discovery is a critical function in type, implemented by Chen et al. [33] for an intelli- pervasive computing that enables ad-hoc associations gent meeting room application, was used to demon- between service providers and consumers to be estab- strate that the SOUPA policy ontology, and its asso- lished [126]. Gaia, CoOL, and SDP have demonstrated ciated algorithms enable to develop intelligent agents that semantic service discovery is better than tradi- that can provide user privacy protection in a pervasive tional, rigid, syntactic service matching. OWL-S, an context-aware environment. excerpt of whose classes is shown in Figure 11, is cur- A policy ontology is also proposed by Paganelli rently the most popular of the available technologies and Giuli within their context-aware framework for for semantic service description. As aforementioned, at home monitoring of patients with chronic diseases though it suffers certain limitations when it comes to [140]. This ontology however addresses the manage- fine-grained service mathing, where priorities on indi- ment of alarm notifications, specifying among others vidual service attributes need to be addressed. Other priority information about which care person should representative semantic service description languages be contacted first, and does not cover privacy and trust include WSML, WSDL-S/SAWSDL, Monolithic DL- aspects. based, and DSD for non-logic-based service IOPE profile [106]. Each of these languages has its own 5.7.1. Analysis strengths and particular application areas, and the cur- As we wish to communicate increasing amounts of rent challenge is how to enhance the expressiveness of (personal) information between services, the topics of service annotations so as to increase the capabilities security, privacy, and access control become more and for searching for candidate services. more critical. To harvest the advantages that Semantic Approximate matching is another interesting topic, Web technologies bring in communicating and sharing as dealing with vague or incomplete descriptions about knowledge across systems, a consensual policy ontol- the functionality of services or user preferences is not ogy is highly desirable in order to capture in a stan- uncommon. Such considerations are not exclusive to dardised manner aspects pertinent to privacy and ac- the pervasive domain, and have attracted significant cessibility, such as who is granted access to which body of research in the semantic matchmaking com- parts of data and for what purpose (e.g., reading or munity [106,70]. editing). The need towards such standardised methods for expressing and exchanging privacy policies, has 5.7. Privacy and Trust been widely acknowledge within the Semantic Web community itself (e.g. ) and is closely related with the Privacy and trust are of paramount concern, if per- quest and recent efforts for formalising the semantics vasive computing applications are to become popu- of provenance information (see Section 6.3). lar outside the laboratory; yet, only a limited number of the reviewed pervasive frameworks have explicitly 5.8. Summary considered these aspects. One such example is the policy-based approach im- This section explicated the use of Semantic Web plemented in CoBrA, where the SOUPA policy ontol- technologies in the domain of pervasive, sensor-driven ogy [36] is used to enforce user privacy. Using this applications. The discussion has been structured along ontology, users can define customised policy rules to the five key requirements, as identified in section 2, permit or forbid different computing entities to access namely: i) conceptual modelling of context (ranging their private information. The policy reasoning algo- from raw sensor data to higher-level context and event rithm uses a DL inference engine and the DL con- abstractions), ii) reasoning about context, iii) uncer- structs of OWL to decide whether an action for access- tainty management, iv) service discovery and v) pri- ing some user private information is permitted. Since it vacy and trust. Besides outlining strengths and weak- is often infeasible to define explicit policy rules for ev- nesses of the proposed approaches, relevant results 32 Ye et alia / Semantically-enriched pervasive sensor-driven systems

Table 5 Overview of Semantic Web technologies use in pervasive applications with respect to key requirements. Requirement SW-empowered approach Added value Further research inquiries Modelling and ontologies for describing sensor facilitate knowledge sharing, reuse upper ontologies; comparative as- representing data, who/where/what/when infor- and exchange; rich expressiveness sessment of existing ontologies context mation, events; rules for higher re- for capturing complex semantic relational expressivity lations Reasoning about DLs reasoning services; rule-based inferring implicit knowledge, con- scalability; combining open- and context reasoning; reasoning with ontolo- sistency checking, reasoning about closed-world reasoning; reasoning gies and rules incomplete knowledge under inconsistency; temporal/real- time reasoning; hybrid reasoning frameworks Uncertainty han- extensions for fuzzy/probabilistic reasoning over imprecise/vague scalability; seamless integration of dling semantics (e.g., Fuzzy DLs, PR- knowledge; reasoning to hypothe- uncertainty; unified handling of dif- OWL); non-standard inferencing ses ferent types of uncertainty (e.g. abduction in DLs ABoxes) Service discovery standardised (DAML-S, OWL-S) semantic discovery and matchmak- scalability; adaptive semantic ser- and custom service-oriented on- ing of services vice composition and execution tologies (e.g. due to service unavailability) Privacy and trust custom policy ontologies facilitate sharing and exchanging of standardised policy/provenance on- privacy policies tologies within the Semantic Web community that have not inquiries and directions. This section analyses open re- been yet explored have been discussed, sketching pos- search issues relevant to temporal features, dynamism, sible directions for further investigations. provenance, and programming that need to be also in- Table 5 summarises the main observations. The use vestigated towards the seamless integration of Seman- of Semantic Web technologies induces a number of tic Web technologies and pervasive computing. straightforward benefits, as a direct result of the ad- vocated explicit semantics and well-defined reason- 6.1. Temporal Features ing services. Issues open to further investigation are to a large extent inherited by challenges that remain The role of temporal information in the modelling open in Semantic Web research too. Efficiency of rea- of sensed data is often simplified in the design of data soning is crucial, as the need to draw complex in- models, systems software, and application APIs. Typi- ferences from millions or billions of pieces of infor- cally, sensor values are recorded in conjunction with a mation in real-time, is not restricted in the domain timestamp. However, despite its universality as a mod- of pervasive systems alone. The seamless integration elling concept, the use of timestamps in isolation im- of open- and closed-world reasoning is another such plicitly restricts a data model to representing only cur- highly desirable feature and subject of active research, rent state; that is, a timestamp has an implicit dual and the same holds for investigations into the practical meaning: it is both the time at which a statement was management of imperfect (uncertain, vague, missing, asserted in the model and the time at which the state- noisy) knowledge. Last but not least, throughout the ment should be interpreted as being true. This distinc- study of SW-based pervasive frameworks it was evi- tion is often unimportant, however the ability to sep- dent that relevant insights often permeated only par- arately annotate data with temporal extent, i.e., a time tially, and in a fragmented manner, the borders be- interval, caters explicitly for the latter case. tween the two research communities. Further to this, the use of both an update timestamp and temporal extent in concert provides a useful facility for modelling both historical and predictive state. 6. Challenging Issues This is often useful, for example: summarising a high volume of sensor readings over a period of time is In the above sections, we have discussed key re- preferable to removing it entirely in the case where quirements in pervasive computing and have analysed data will later be analysed offline, or representing a the benefits and potentials that are induced by using person’s predicted future locations based on calendar Semantic Web technologies, along with future research data. Modelling time as an orthogonal concern allows Ye et alia / Semantically-enriched pervasive sensor-driven systems 33 such cases to be handled without requiring special infor the validity of named graphs based on Elmasari et dicators in a data model to indicate that information al.’s Time Index [67]. has been treated thusly. To facilitate querying of temporal RDF, Tappo- Despite strong vocabulary support for represent- let et al. introduce a syntactic sugar extension to ing temporal features, as discussed in section 5.2, the SPARQL to facilitate the querying of graphs valid at triple-based nature of the RDF model is ill-suited to a certain time point [191]. McBride et al. [118] at- the representation of data’s temporal properties, where tach temporal extent to objects, upon which they pro- the requirement is to annotate sets of triples. At least vide a syntactic sugar in query language that maps three possible strategies for overcoming this limitation to SPARQL. Rodriguez et al. introduce a scheme are available to data modellers: using RDF’s reification for storing, indexing and querying temporal data de- vocabulary, externally generating identifiers for state- signed for sensor networks [2]. Perry et al. [148] in- ments, or use a non-standard extension to the RDF troduce the syntax and semantics of an extension to model. SPARQL, called SPARQL-ST, designed for the exe- To utilise RDF’s reification vocabulary, each triple cution of spatiotemporal queries. O’Connor et al. de- is expanded as a ‘reification quad’12, with the resul- velop a lightweight temporal model to encode data tant statement identifier associated with temporal in- based on the valid-time dimension, upon which they formation. Generating a statement identifier externally develop extensions to their SWRL-based OWL query follows similarly, but avoids the introduction of redun- language SQWRL [137,138]. Their query library sup- dant information into the model at the expense of reports Allen’s relations and provides functions for grouping query results such that the filters first, first-n, quiring non-standard tooling to generate and resolve last, last-n, and nth can be applied within a query. such identifiers. In addition to solutions that are solely based on tem- Temporal extensions to RDF, all notionally based on poral extensions of Semantic Web technologies, hybrid expanding the triple model to a quad model, have been frameworks that combine ontologies with temporally- explored in the literature. Lilis et al. [113] introduce aware formalisms have also been investigated. In the Multidimensional RDF, an extension to RDF designed ambient computing framework proposed by Patkos et to express the temporal semantics of a collection of al. [145] for example, the inherent temporal reasoning cultural artefacts. Time is used as a contextual specifier capabilities of the Event Calculus [107] are utilised. to control whether or not an RDF triple should be con- Rule-based reasoning is used, on top of context mod- sidered to be present in a graph at a given time point or elling ontologies, to infer complex contexts from raw interval. Gutierrez et al. [81] provide the semantics for context data. In parallel, causality reasoning is em- temporal RDF graphs, introducing the notion of a tem- ployed to capture and reason over preconditions and poral triple, an RDF triple with a temporal label, and effects of actions and events, based on the Event Cal- a temporal graph made from a set of temporal triples. culus theory. A main advantage, in comparison to plain The authors present the concepts of graph slices and rule-based approaches, is the inherent notion of time graph snapshots, which allow for the description of the in the Event Calculus that allows to establish a linear collection of triples that hold during or at a given in- time ordering and hence infer in which intervals cer- terval or instant. This foundational work in modelling tain conclusions hold, while re-evaluating event pat- dynamic data using RDF, influences the design of the terns as time progresses. The situation awareness ar- temporal component of our sensor data model. chitecture presented in [8], is another example where Based on the notion of temporally-extended RDF, different formalisms and available reasoners have been Pugliese et al. [153] introduce the tGRIN index struc- combined to enable efficient inference over data that ture that builds a specialised index for storing temporal change over time. Sensor observations are first ag- RDF in a relational database based on temporal as well gregated by means of if-then rules, and subsequently as the structural closeness of triples, while Tappolet et fed to the semantic interpretation layer, where a DL al. [191] present a method for building a meta-index reasoner is used for situation assessment. The SCEP frameworks described in section 5.3.7, coupling the inherent real-time reasoning capabilities of CEP engines 12The term ‘reification quad’ refers to a set of 4 triples that associate an identifier with the type rdf:Statement, and three prop- with rich ontological semantics, are yet another exam- erties, rdf:subject, rdf:predicate, and rdf:object ple of investigations towards the efficient handling of with values corresponding to the original triple. temporal semantics. 34 Ye et alia / Semantically-enriched pervasive sensor-driven systems

The aforementioned indicate a significant gap be- static data as it is generated, adding the inferred knowl- tween the currently available Semantic Web technolo- edge directly to the model, while performing reason- gies and the need for native temporal data modelling ing on highly dynamic data only at the point where an and reasoning. Relevant suggestions exist in research application query is executed, restricting reasoning to prototype form, but considerable efforts are required the smallest amount of volatile data that will produce a before achieving standardised solutions that will in correct answer. Based solely on data’s temporal prop- turn ensure necessary tool support for efficient reason- erties, it may be possible to devise a general scheme ing over temporal data and management of temporal to partition, reason on, and integrate data over several queries. Within this quest, hybrid frameworks present stages so as to optimise reasoning. appealing features, as through the combination of different formalisms and reasoners, extended representa- 6.3. Provenance tional and reasoning capabilities can be achieved; the seamless and scalable integration of the heterogeneous Many of the issues above can be subsumed un- modules inevitably poses its own challenges. der the notion of provenance, i.e. the capture, representation and manipulation of knowledge relevant 6.2. Dynamism to data creation, ownership, transformation and other ‘lifecycle’ issues. Provenance appears in many guises. Distinct from the need to model the temporal fea- The Dublin Core vocabulary [55] is perhaps the best- tures of dynamic data are the challenges related stor- known, providing terms for asserting authorship and ing, reasoning over, and accessing large volumes of other property rights over digital objects. More re- highly-dynamic, distributed information. cently a task group of the World Wide Web Con- Whether a centralised or decentralised setup is sortium has been standardising the Open Provenance adopted, as the volume of sensor data increases, it be- Model [127] for asserting more general provenance comes infeasible to store a permanent record of gener- metadata. ated states. A typical solution might be to discard the Sensor-driven systems are more directly affected oldest data from memory when the maximum capacity by data provenance than programs in many other do- is approached. However, it does not necessarily follow mains. A sensor system must make decisions using in- that the oldest data stored is the least useful. One possible solution might be to require that the data model put data that is known to be inherently noisy, impre- captures properties describing data dynamics such that cise, inaccurate, untimely and infrequent, with the de- policies may be defined to prioritise the discard of data gree of each source of error perhaps varying signifi- that is asserted frequently but has a slow rate of change cantly over time. An immediate consequence of this (e.g., ambient temperature) over data that is frequently is that sensor-driven systems cannot be directly con- asserted but changes rapidly (e.g., a user’s coordinate nected to their input data streams, or respond directly location) or pseudo-static data that is infrequently as- to them: to do so is to allow the system to respond to serted (e.g., building layouts) [178]. transient noise. Another way of putting this is that in- Data dynamics also play a role in determining the dividual data elements are evidence of fact rather than types of reasoning strategies that should be adopted. being facts themselves, and must be fused with other There is a need to track derivations such that it may be data (from the same or different sources) to build a determined when an inference no longer holds due to consensus of the state of the environment being sensed. modifications to or invalidation of the data upon which While sensor fusion is commonplace in much of it is predicated. Structuring the data model so that data engineering, many of the approaches make strong as- is temporally qualified (and hence inferences are also sumptions about the nature of the data streams: for ex- temporally qualified) forms part of a solution to this ample they are from homogeneous sensors looking at issue, but requires the incorporation of the appropri- the same phenomenon and with well-known sampling ate temporal semantics within existing reasoning tech- frequencies. Such techniques do not generalise well nologies. to semantically-enriched systems with highly hetero- The process of reasoning over a large data model geneous data sources processed using symbolic rea- can be a performance bottleneck, however knowledge soning. Conversely, homogeneous data can be handled about the dynamics of data can inform an appropriate easily within reasoners, allowing semantic technology reasoning strategy. For example, choosing to reason on to encompass many sensor fusion tasks. Ye et alia / Semantically-enriched pervasive sensor-driven systems 35

Reasoning over sensor data is affected by a num- 6.4. Programming ber of provenance factors of data streams and individual data. A trivial example is that the reliability of a Lastly we come to the practicalities of program- temperature estimated from data that is several hours ming. old may be assumed, all other things being equal, to Semantic structures like RDF allow rich encodings be less than that inferred from data collected within of data. Coupled with OWL, SPARQL, ontological the previous minute. Sensor types (and even individ- and other reasoners, it is possible to answer complex ual sensors) give rise to data with given provenance queries by traversing the knowledge graph. The emer- in terms of the observation frequency, precision and gence of standard, re-usable reasoners significantly the like, which can be captured generically or using simplifies the use of semantic technologies within pro- special-purpose markup languages like SensorML (see grams. section 5.1). The provenance here is associated with The integration of these tools remains superficial, the sensor but attaches to the data (stream or items) however. From a programming perspective, a knowl- produced by the sensor. edge graph is simply a collection of edges labelled More generally, a system may be interested in the with strings and URIs, possibly with some additional entire lifecycle of a data stream including the transfor- typing at the endpoints supplied by XML Schemata, mations that have been applied to it. Suppose again we and with some structure on the relationships provided are receiving a stream of temperature values which we by the accompanying ontologies (if any). The gener- intend to use in some decision process. It may matter ality of these structures forces programmers to pro- whether those values are ‘raw’ (and subject to raw sen- vide any additional machinery by hand, without the sor noise), and have been processed to remove obvi- assistance of a compiler, type system of other tooling with which we are familiar in the building of complex ous outliers, or have been smoothed using an a priori software systems. In an effort to simplify the integra- or learned smoothing function. It is therefore vital for tion of Semantic Web technologies into existing soft- systems where data comes with provenance that this ware architectures and languages, a plethora of Ap- provenance is manipulated and maintained along the plication Programming Interfaces (APIs) for working data pathway. For many scientific systems the statisti- with RDF/OWL ontologies have been developed, such cal properties of a data stream are as important as the as Jena API [99], OWLAPI [90], dotNetRDF [62], data itself. Extensive in-system processing of data is RDFReactor [207] and Sesame [171]. not necessarily a problem, if it is clear to the end-user Use of semantic technology implies considerable fa- that such processing has occurred and to what effect miliarity with the tools of XML, notably namespaces, – otherwise any further statistical calculations are ren- that obscure rather than illuminate the underlying in- dered suspect. formation being encoded. Data so encoded is not held Provenance is increasingly recognised as a crucial in this form in memory, and so must be translated element of the Semantic Web, as it enables inferences for storage and exchange: a non-trivial process in the to be drawn conditional to whether information should presence of shared sub-structures and update. State-of- be trusted and how it should be reused and integrated the-art ontology editors, such as Protégé [136], Top- with other diverse information sources. The lack, how- Braid Composer [198], and NeOn Toolkit [134] offer ever, of a standard model for capturing, interchanging comprehensive support for developing and validating and reasoning provenance metadata is a significant im- RDF/OWL ontologies. Moreover, scalable and query pediment to realising applications where the trustwor- efficient RDF repositories, such as OWLIM [18], Al- thiness and the quality of the statements is at issue. The legroGraph [3], and OpenLink Virtuoso [68], and as Provenance Working Group [151] is an ongoing effort well as, database-to-RDF mapping tools such as D2RQ towards the standardisation of an interchange core lan- [86,66], enable data to be exposed and shared on the guage (PROV Ontology [149]) for publishing and ac- Web according to the principles of Linked Data [19]. cessing provenance metadata, drawing on existing vo- Entities represented within a knowledge graph may cabularies and ontologies [127,150,152]. In parallel, be classified by the ontology according to the informa- recent research studies have proposed approaches that tion present about them – classifying a person as an enable the handling of provenance while using and rea- employee given the presence of an employee number soning about information and resources in open and and/or a line manager within the same organisation, for collaborative environments [22,139,196]. example. This is superficially the same as a program- 36 Ye et alia / Semantically-enriched pervasive sensor-driven systems ming language type system, with the difference that an programming with semantic structures. Sensor-driven entity’s class may change unpredictably and at unpre- systems add the necessity to embrace the noise and er- dictable times through the actions of other agents on rors inherent in data streams, and to capture and make the knowledge graph, and it can be hard to predict ex- decisions that propagate this uncertainty throughout actly which such changes will have such an effect. This the system. These are not issues that mainstream pro- destabilises a program’s view of the knowledge in the gramming languages provide structures for, although graph. it is possible to build programs that (for example) Use of a reasoner sits outside the programming lan- maintain the error bars on results derived from pro- guage, in the same as does SQL when accessing a cessing inputs that themselves have error bars. Com- database: a SPARQL query is a string that returns ar- bining programming with uncertainty and program- rays of other strings, and is not type-checked or manip- ming with data that adhere to an ontological struc- ulated using dedicated programming constructs. This ture, seems to be two new frontiers for programming complicates the formation and checking of complex systems design opened-up by semantically-enriched queries, again from a lack of supporting tooling. Some sensor-driven systems. of the aforementioned ontology editors provide only a Perhaps the chief impediment to such designs comes limited support for advanced query formulation, e.g., from the challenged nature of the platforms them- type-checking. selves. Sensor-driven systems place a significant amount Finally, the Semantic Web may present steep learn- of their functionality on devices with extremely con- ing and commitment curves, particularly for novices strained memory, computation, and communication coming from other research areas. In order to perform capabilities, which are not straightforwardly subject even simple tasks, a developer must master a wide to Moore’s-law-driven improvements in performance. range of perhaps unfamiliar technologies. Moreover, A naïve deployment of Semantic Web technologies to an organisation must commit to these technologies and such platforms is doomed to failure, but there seems their concomitant costs ahead of time. This raises a to be no a priori reason why versions optimised for barrier to deployment by increasing the risk that a sys- restricted domains might not be possible, and might tem may not generate the expected benefits while in- not interoperate easily with the wider universe of web- curring a substantial up-front cost. enabled components. As with any technology, the decision to use Seman- tic Web technologies, does not come without a price, and one needs to ascertain sufficient value on its ad- 7. Discussion and Concluding Remarks vantages – open, standards-based representation, easy exchange and integration – to make it worthwhile. It In this paper, we reviewed the landscape of the apis undoubtedly attractive to be able to define a struc- plications of Semantic Web technologies to the domain ture for knowledge that exactly matches a chosen sub- of pervasive, adaptive, sensor-driven systems. Many of domain, to describe the richness of this structure, and the features underpinning the Semantic Web, and in to have it compose cleanly with other such descrip- particular, the ability to formally capture and reason tions of complementary sub-domains defined indepen- over rich, semantic interconnections between data, in dently – and to be able to exchange all this knowledge an open and extensible way, are well-suited to perva- with anyone on the web. But this flexibility comes with sive computing. At the same time however, a number a cost and (often) no obvious immediate, high-value of other important aspects, such as lack of native sup- benefits. port for representing and reasoning over temporal or In many ways, these issues are an inevitable conse- imperfect knowledge, remain under-articulated. quence of the differences in the application domains of The key benefits of Semantic Web technologies semantic technology and programming languages. The are rather straightforward in terms of modelling and former addresses the open, extensible, scalable, dis- reasoning about context. As afore-described, ontolo- tributed mark-up of data. The latter addresses almost gies have been proposed for describing the four Ws exactly opposite issues, focusing on close specification (when, where, what, who) that characterise contextual of algorithms and data structures to share common data knowledge, complex events and sensor data, facili- and functionality. tating the unambiguous sharing and understanding of Sensor-driven systems add few, if any, specific knowledge across heterogeneous and distributed plat- points to this general discussion: the issues apply to all forms, devices and services. What still remains un- Ye et alia / Semantically-enriched pervasive sensor-driven systems 37 addressed, acknowledging the different world-views model of the Semantic Web seem eminently possible that different ontologies inevitably serve, is the defini- and deserve future exploration. tion of commonly-agreed (possibly upper) ontologies Summing up, the ability to exchange models and that would enable the standardised description of perti- data, to reason openly, to capture an extending set of nent context aspects; the conceptual overlaps between data and metadata, and to interact with other web- the proposed ontologies, underline further this need. enabled elements, all encourage the view that bas- Moreover, since not all of the provided modelling ca- ing future pervasive and sensor-driven systems around pabilities have been directly applied in pervasive appli- these technologies would lead to significant improve- cations (e.g. modelling of composite events and their ments in interoperability and semantic clarity – if the dependencies that many of the event ontologies sup- disparate elements can be integrated into a framework port), it is time to systematically assess their applica- appropriate for system developers. Such an integra- bility, possibilities for their combined use and/or (par- tion would allow innovation to proceed more rapidly tial) alignment, as well as aspects that require a more and soundly, bringing Weiser’s vision of seamless, elaborate coverage. integrated pervasive and sensor systems significantly In parallel, the deployment of Semantic Web tech- closer. nologies has demonstrated the intrinsic relation between automated reasoning and the high-level interpretation of context data, where the integration of References structured domain knowledge is a prerequisite. Besides useful insights on the need to combine ontologies with [1] A. Agostini, C. Bettini, and D. Riboni. Hybrid reasoning in rules, the reviewed literature sketches a number of de- the care middleware for context awareness. Int. J. Web Eng. sirable, yet highly challenging questions. Prominent Technol., 5(1):3–23, May 2009. ones include the need to provide native support for rep- [2] Y. L. Alejandro Rodríguez, Robert McGrath and J. Myers. Semantic management of streaming data. In Proceedings of resenting and reasoning over temporal knowledge, in- the 2nd International Workshop on Semantic Sensor Networks corporating provenance into reasoning, and managing (SSN09), volume 522 of CEUR Workshop Proceedings, pages uncertainty. An indispensable requirement, underlying 80–95, 2009. all of aforementioned research directions, is computa- [3] AllegroGraph RDFStore Web 3.0’s Database. http:// tional efficiency. Ensuring reasoning efficiency is cru- www.franz.com/agraph/allegrograph/. [4] J. F. Allen. Towards a general theory of action and time. Ar- cial, as the need to draw inferences in real-time, from tificial Intelligence, 23(2):123–154, 1984. millions or billions of pieces of information, in an [5] D. Anicic, P. Fodor, S. Rudolph, and N. Stojanovic. Ep- open pervasive environment, already challenges exist- sparql: a unified language for event processing and stream ing reasoning engines, even though considering only reasoning. In Proceedings of the 20th international confer- deductive inference over crisp, consistent knowledge ence on World wide web, WWW ’11, pages 635–644, Hyder- abad, India, 2011. ACM. bases. [6] D. Anicic, S. Rudolph, P. Fodor, and N. Stojanovic. Stream Subject to further enquiry is also the overall inte- reasoning and complex event processing in etalis. Semantic gration of Semantic Web technologies within program- Web - Interoperability, Usability, Applicability, 1:1–5, 2011. ming frameworks suitable for software engineering in- [7] G. Antoniou and F. v. Harmelen. A Semantic Web Primer, 2nd the-large. The Semantic Web at present is essentially Edition (Cooperative Information Systems). The MIT Press, 2 edition, 2008. a collection of fragments lacking a whole. This is per- [8] F. Baader, A. Bauer, P. Baumgartner, A. Cregan, A. Gabal- haps an inevitable consequence of an architecture de- don, K. Ji, K. Lee, D. Rajaratnam, and R. Schwitter. A novel signed for such a broad spectrum of application do- architecture for situation awareness systems. In TABLEAUX, mains, but it nevertheless increases the risks and costs pages 77–92, 2009. associated with applying the technologies to pervasive [9] F. Baader, D. Calvanese, D. L. McGuinness, D. Nardi, and P. F. Patel-Schneider, editors. The Description Logic Hand- systems. This is especially the case when the target book: Theory, Implementation, and Applications. Cambridge platforms for much of the functionality are challenged University Press, 2003. in terms of their memory, computation and communi- [10] F. Baader and U. Sattler. An overview of tableau algorithms cations capabilities. It is important to remember, how- for description logics. Studia Logica, 69(1):5–40, 2001. ever, that the Semantic Web is essentially a tool of [11] A. Bandara, T. Payne, D. D. Roure, N. Gibbins, and T. Lewis. Semantic resource matching for pervasive environments: The modelling and exchange, and only secondarily a tool of approach and its evaluation. Technical report, Faculty of implementation: compact representations of informa- Physical and Applied Science, University of Southampton, tion and reasoning that comply to the underlying meta- 2008. 38 Ye et alia / Semantically-enriched pervasive sensor-driven systems

[12] N. Baumgartner and W. Retschitzegger. A survey of upper Foundations of Software Technology and Theoretical Com- ontologies for situation awareness. In In Proceedings of the puter Science, pages 87–93, London, UK, 2000. 4th IASTED International Conference on Knowledge Sharing [28] M. Burstein, J. Hobbs, O. Lassila, D. Martin, S. McIl- and Collaborative Engineering, pages 1–9, St. Thomas, US raith, S. Narayanan, M. Paolucci, T. Payne, K. Sycara, Virgin Islands, November 2006. and H. Zeng. DAML-S Draft Release (May 2001). [13] S. Ben Mokhtar, A. Kaul, N. Georgantas, and V. Issarny. Ef- http://www.daml.org/services/daml-s/2001/05/, 2001. ficient semantic service discovery in pervasive computing en- [29] M. H. Burstein, J. R. Hobbs, O. Lassila, D. Martin, D. V. Mc- vironments. In Proceedings of the ACM/IFIP/USENIX 2006 Dermott, S. A. McIlraith, S. Narayanan, M. Paolucci, T. R. International Conference on Middleware, Middleware ’06, Payne, and K. P. Sycara. DAML-S: Web Service Description pages 240–259, Melbourne, Australia, 2006. Springer-Verlag for the Semantic Web. In Proceedings of the First Interna- New York, Inc. tional Semantic Web Conference on The Semantic Web, ISWC [14] T. Berners-Lee, J. Hendler, and O. Lassila. The semantic web. ’02, pages 348–363, London, UK, 2002. Springer-Verlag. Scientific American, 284(5):34–43, May 2001. [30] R. N. Carvalho, K. B. Laskey, and P. C. G. da Costa. PR-OWL [15] C. Bettini, O. Brdiczka, K. Henricksen, J. Indulska, D. Nick- 2.0 - Bridging the gap to OWL semantics. In URSW, pages las, A. Ranganathan, and D. Riboni. A survey of context mod- 73–84, 2010. elling and reasoning techniques. Pervasive and Mobile Com- [31] G. Castellano, S. D. Villalba, and A. Camurri. Recognising puting, 6(2):161–180, 2010. human emotions from body movement and gesture dynamics. [16] A. Bikakis and G. Antoniou. Contextual argumentation in In ACII ’07: Proceedings of the 2nd international conference ambient intelligence. In Proceedings of the 10th International on Affective Computing and Intelligent Interaction, pages 71– Conference on Logic Programming and Nonmonotonic Rea- 82, Lisbon, Portugal, 2007. Springer-Verlag, Berlin, Heidel- soning (LPNMR ’09), pages 30–43, Potsdam, Germany, 2009. berg. Springer-Verlag. [32] H. Chen, T. Finin, and A. Joshi. An Ontology for Context- [17] A. Bikakis, T. Patkos, G. Antoniou, and D. Plexousakis. A Aware Pervasive Computing Environments. Special Issue on Survey of Semantics-based Approaches for Context Reason- Ontologies for Distributed Systems, Knowledge Engineering ing in Ambient Intelligence. In Proceedings of the Work- Review, 18(3):197–207, May 2003. shop Artificial Intelligence Methods for Ambient Intelligence, [33] H. Chen, T. Finin, and A. Joshi. A Pervasive Computing pages 15–24, 2007. Ontology for User Privacy Protection in the Context Broker [18] B. Bishop, A. Kiryakov, D. Ognyanoff, I. Peikov, Z. Tashev, Architecture. Technical Report TR-CS-04-08, University of and R. Velkov. OWLIM: A family of scalable semantic repos- Maryland, Baltimore County, July 2004. itories. Semant. web, 2(1):33–42, Jan. 2011. [34] H. Chen, T. Finin, and A. Joshi. Semantic web in the context [19] C. Bizer, T. Heath, and T. Berners-Lee. Linked Data - The broker architecture. In PERCOM ’04: Proceedings of the Sec- Story So Far. International Journal on Semantic Web and ond IEEE International Conference on Pervasive Computing Information Systems (IJSWIS), 5(3):1–22, MarMar 2009. and Communications (PerCom’04), pages 277 – 286, Wash- [20] F. Bobillo and U. Straccia. Fuzzy ontology representation ington, DC, USA, 2004. IEEE Computer Society. using OWL 2. Int. J. Approx. Reasoning, 52(7):1073–1094, [35] H. Chen, T. Finin, and A. Joshi. Ontologies for Agents: The- 2011. ory and Experiences, chapter of The SOUPA Ontology for [21] J. Bohn, F. C. Gartner, and H. Vogt. Dependability Issues of Pervasive Computing. Whitestein Series in Software Agent Pervasive Computing in a Healthcare Environment. In first Technologies. Springer, 2005. International Conference on Security in Pervasive Comput- [36] H. Chen, F. Perich, D. Chakraborty, T. Finin, and A. Joshi. In- ing, Boppard, Germany, 2003. telligent agents meet semantic web in a smart meeting room. [22] P. A. Bonatti, A. Hogan, A. Polleres, and L. Sauro. Robust In Proceedings of the Third International Joint Conference on and scalable linked data reasoning incorporating provenance Autonomous Agents and Multiagent Systems - Volume 2, AA- and trust annotations. J. Web Sem., 9(2):165–201, 2011. MAS ’04, pages 854–861, New York, New York, 2004. IEEE [23] S. Borgo, N. Guarino, and C. Masolo. A pointless theory of Computer Society. space based on strong connection and congruence. In L. C. [37] H. Chen, F. Perich, T. Finin, and A. Joshi. SOUPA: Standard Aiello, J. Doyle, and S. Shapiro, editors, KR’96: Principles Ontology for Ubiquitous and Pervasive Applications. In First of Knowledge Representation and Reasoning, pages 220–229. Annual International Conference on Mobile and Ubiquitous Morgan Kaufmann, San Francisco, California, 1996. Systems, Boston, MA, USA, August 2004. [24] G. Borriello, V. Stanford, C. Narayanaswami, and W. Men- [38] H. L. Chen. An Intelligent Broker Architecture for pervasive ning. Guest editors’ introduction: Pervasive computing in context-aware systems. PhD thesis, University of Maryland, healthcare. IEEE Pervasive Computing, 6(1):17–19, Jan. 2004. 2007. [39] L. Chen, C. Nugent, M. Mulvenna, D. Finlay, and X. Hong. [25] B. Bouchard and A. Bouzouane. A key hole plan recognition Semantic smart homes: Towards knowledge rich assisted liv- model for alzheimer’s patients: First results. Applied Artificial ing environments. Intelligent Patient Management, 189:279– Intelligence, 22:1–34, 2007. 296, 2009. [26] A. Bulling, J. A. Ward, H. Gellersen, and G. Triöster. Ro- [40] L. Chen and C. D. Nugent. Ontology-based activity recogni- bust recognition of reading activity in transit using wearable tion in intelligent pervasive environments. International Jour- electrooculography. In Proceedings of Pervasive 2008, pages nal of Web Information Systems, 5(4):410–430, 2009. 19–37. Springer Berlin / Heidelberg, 2008. [41] L. Chen, C. D. Nugent, and H. Wang. A knowledge-driven [27] P. Buneman, S. Khanna, and W. C.Tan. Data provenance: approach to activity recognition in smart homes. IEEE Trans- Some basic issues. In Proceedings of the 20th Conference on actions on Knowledge and Data Engineering, 2011. To ap- Ye et alia / Semantically-enriched pervasive sensor-driven systems 39

pear. information. In R. Meersman, Z. Tari, and S. M. Stevens, [42] D. Choujaa and N. Dulay. Activity inference through se- editors, DS-8, volume 138 of IFIP Conference Proceedings, quence alignment. In LoCA ’09: Proceedings of the 4th In- pages 351–369. Kluwer, 1999. ternational Symposium on location and Context Awareness, [57] A. K. Dey. Understanding and Using Context. Personal and pages 19–36, Tokyo, Japan, 2009. Springer-Verlag. Ubiquitous Computing, 5(1):4–7, 2001. [43] E. Christopoulou, C. Goumopoulos, I. Zaharakis, and [58] Z. Ding, Y. Peng, R. Pan, Z. Ding, Y. Peng, and R. Pan. A. Kameas. An ontology-based conceptual model for com- BayesOWL: Uncertainty modeling in semantic web ontolo- posing context-aware applications. In Proceedings of the First gies. Soft Computing in Ontologies and Semantic Web, pages International Workshop on Advanced Context Modelling, 3–29, 2006. Reasoning And Management (in UbiComp 2004), September [59] R. Dividino, S. Sizov, S. Staab, and B. Schueler. Querying 2004. for provenance, trust, uncertainty and other meta knowledge [44] X. Chu and R. Buyya. Service oriented sensor web. In Sensor in rdf. Web Semant., 7:204–219, September 2009. Networks and Configuration, pages 51–74. Springer Berlin [60] S. Dobson. Leveraging the subtleties of location. In Proceed- Heidelberg, 2007. ings of the 2005 joint conference on Smart objects and am- [45] A. Ciaramella, M. G. C. A. Cimino, F. Marcelloni, and bient intelligence: innovative context-aware services: usages U. Straccia. Combining fuzzy logic and semantic web to en- and technologies, sOc-EUSAI ’05, pages 189–193, Grenoble, able situation-awareness in service recommendation. In Pro- France, 2005. ACM. ceedings of the 21st international conference on Database [61] M. Doerr. The CIDOC conceptual reference module: an onto- and expert systems applications: Part I, DEXA’10, pages 31– logical approach to semantic interoperability of metadata. AI 45, Berlin, Heidelberg, 2010. Springer-Verlag. Mag., 24(3):75–92, Sept. 2003. [46] A. G. Cohn, B. Bennett, J. Gooday, and N. M. Gotts. Qualita- [62] dotNetRDF - Semantic Web/RDF Library for C#/.Net. tive spatial representation and reasoning with the region con- http://www.dotnetrdf.org/. nection calculus. Geoinformatica, 1(3):275–316, Oct. 1997. [63] J. Du, G. Qi, Y.-D. Shen, and J. Z. Pan. Towards practical [47] M. Compton, P. Barnaghi, L. Bermudez, R. Garcia-Castro, ABox abduction in large OWL DL ontologies. In Proceed- O. Corcho, S. Cox, J. Graybeal, M. Hauswirth, C. Henson, ings of the Twenty-Fifth AAAI Conference on Artificial Intel- A. Herzog, V. Huang, K. Janowicz, W. D. Kelsey, D. L. ligence, AAAI 2011, pages 1160–1165, San Francisco, Cali- Phuoc, L. Lefort, M. Leggieri, H. Neuhaus, A. Nikolov, fornia, USA, 2011. AAAI Press. K. Page, A. Passant, A. Sheth, and K. Taylor. The SSN Ontol- [64] DOLCE+DnS Ultralite (DUL) ontology. http://www. ogy of the W3C Semantic Sensor Network Incubator Group. loa.istc.cnr.it/ontologies/DUL.owl. Web Semantics: Science, Services and Agents on the World [65] E. Dumbill. Finding friends with XML Wide Web, 0(0), 2012. and RDF: FOAF, 2002. http://www- [48] M. Compton, C. Henson, L. Lefort, H. Neuhaus, and 106.ibm.com/developerworks/xml/library/x-foaf.html. A. Sheth. A survey of the semantic specification of sensors. In [66] V. Eisenberg and Y. Kanza. D2rq/update: updating relational Proceedings of the 2nd International Workshop on Semantic data via virtual rdf. In Proceedings of the 21st international Sensor Networks (SSN09), pages 17–32, 2009. conference companion on World Wide Web, WWW ’12 Com- [49] D. J. Cook. How smart is your home? Science, 335:1579– panion, pages 497–498, Lyon, France, 2012. ACM. 1581, Mar. 2012. [67] R. Elmasri, G. T. J. Wuu, and Y.-J. Kim. The time index— [50] D. J. Cook, J. C. Augusto, and V. R. Jakkula. Ambient intelli- an access structure for temporal data. In Proceedings of the gence: Technologies, applications, and opportunities. Perva- sixteenth international conference on Very large databases, sive and Mobile Computing, 5(4):277–298, August 2009. pages 1–12, Brisbane, Australia, 1990. Morgan Kaufmann [51] D. J. Cook and S. K. Das. How smart are our environments? Publishers Inc. an updated look at the state of the art. Pervasive Mobile Com- [68] O. Erling. Virtuoso, a Hybrid RDBMS/Graph Column Store. puting, 3(2):53–73, 2007. IEEE Data Eng. Bull., 35(1):3–8, 2012. [52] P. C. Costa, K. B. Laskey, and K. J. Laskey. Uncertainty rea- [69] K. Farkas, H.-P. Labs, J. Heidemann, and L. Iftode. Intelli- soning for the Semantic Web I. chapter PR-OWL: A Bayesian gent transportation and pervasive computing. Pervasive Com- Ontology Language for the Semantic Web, pages 88–107. puting, 5:18–19, 2006. Springer-Verlag, Berlin, Heidelberg, 2008. [70] G. Fenza, V. Loia, and S. Senatore. A hybrid approach to se- [53] R. Cunningham and V. Cahill. System support for smart cars: mantic web services matchmaking. Int. J. Approx. Reasoning, requirements and research directions. In Proceedings of the 48(3):808–828, 2008. 9th workshop on ACM SIGOPS European workshop: beyond [71] Davies and Vitiello’s relationship vocabulary for FOAF. the PC: new challenges for the operating system, EW 9, pages http://vocab.org/relationship. 159–164, Kolding, Denmark, 2000. ACM. [72] D. Franklin. Cooperating with people: the intelligent class- [54] S. Dasiopoulou and I. Kompatsiaris. Trends and issues in de- room. In Proceedings of the fifteenth national/tenth con- scription logics frameworks for image interpretation. In Pro- ference on Artificial intelligence/Innovative applications of ceedings of the 6th Hellenic conference on Artificial Intel- artificial intelligence, AAAI ’98/IAAI ’98, pages 555–560, ligence: theories, models and applications, SETN’10, pages Madison, Wisconsin, United States, 1998. American Associ- 61–70, Athens, Greece, 2010. Springer-Verlag. ation for Artificial Intelligence. [55] Dublin Core Metadata Initiative. http://dublincore. [73] A. Gangemi and P. Mika. Understanding the semantic web org. through descriptions and situations. In Proceedings of the In- [56] S. Decker, M. Erdmann, D. Fensel, and R. Studer. Ontobro- ternational Conference on Ontologies, Databases and Appli- ker: Ontology based access to distributed and semi-structured cations of SEmantics, pages 689–706, 2003. 40 Ye et alia / Semantically-enriched pervasive sensor-driven systems

[74] J. Gómez-Romero, M. A. Patricio, J. García, and J. M. biCom ’99: Proceedings of the 5th annual ACM/IEEE inter- Molina. Context-based reasoning using ontologies to adapt national conference on Mobile computing and networking, visual tracking in surveillance. In Proceedings of the 2009 pages 249–255, New York, NY, USA, 1999. ACM. Sixth IEEE International Conference on Advanced Video and [90] M. Horridge and S. Bechhofer. The OWL API: A Java API Signal Based Surveillance, AVSS ’09, pages 226–231, Wash- for OWL ontologies. Semant. web, 2(1):11–21, Jan. 2011. ington, DC, USA, 2009. IEEE Computer Society. [91] I. Horrocks and P. F. Patel-Schneider. KR and reasoning on [75] J. Gómez-Romero, M. A. Patricio, J. García, and J. M. the semantic web: OWL. In J. Domingue, D. Fensel, and J. A. Molina. Ontology-based context representation and reasoning Hendler, editors, Handbook of Semantic Web Technologies, for object tracking and scene interpretation in video. Expert chapter 9, pages 365–398. Springer, 2011. Syst. Appl., 38(6):7494–7510, 2011. [92] I. Horrocks, P. F. Patel-Schneider, H. Boley, S. Tabet, [76] C. Goumopoulos, A. D. Kameas, and A. Cassells. An B. Grosof, and M. Dean. SWRL: A Semantic Web Rule Lan- ontology-driven system architecture for precision agriculture guage Combining OWL and RuleML. Technical report, Na- applications. IJMSO, 4(1/2):72–84, 2009. tional Research Council of Canada, Network Inference, and [77] B. C. Grau, I. Horrocks, B. Motik, B. Parsia, P. Patel- Stanford University, May 2004. Schneider, and U. Sattler. OWL 2: The Next Step for OWL. [93] I. Horrocks, P. F. Patel-Schneider, and F. van Harmelen. From Web Semantics: Science, Services and Agents on the World SHIQ and RDF to OWL: the making of a Web Ontology Lan- Wide Web, 6(4):309–322, October 2008. guage. J. Web Sem., 1(1):7–26, 2003. [78] B. N. Grosof, I. Horrocks, R. Volz, and S. Decker. Description [94] S. Hossein, S. Hedal, and A. Mendez-Vasquez. Sensory logic programs: combining logic programs with description data set description language specification. Technical Report logic. In WWW, pages 48–57, 2003. SDDL_Specification_v1.0, University of Florida, 2009. [79] T. Gu, H. K. Pung, and D. Q. Zhang. A service-oriented mid- [95] International Standards Organisation (ISO). ISO-8601: Data dleware for building context-aware services. J. Netw. Comput. elements and interchange formats – Information interchange Appl., 28(1):1–18, Jan. 2005. – Representation of dates and times. ISO, Geneva, Switzer- [80] T. Gu, X. H. Wang, H. K. Pung, and D. Q. Zhang. land, 2004. An Ontology-based Context Model in Intelligent Environ- [96] International organization for standardization, for FDIS ments. In Proceedings of the Communication Networks and 19115 geographic information – metadata. http://www. Distributed Systems Modeling and Simulation Conference ncits.org/ref-docs/FDIS_19115.pdf. (CNDS 2004), pages 270–275, January 2004. [97] V. R. Jakkula, A. S. Crandall, and D. J. Cook. Knowledge [81] C. Gutierrez, C. A. Hurtado, and A. Vaisman. Introducing discovery in entity based smart environment resident data us- time into rdf. IEEE Transactions on Knowledge and Data ing temporal relation based data mining. In ICDMW ’07: Engineering, 19(2):207–218, 2007. Proceedings of the Seventh IEEE International Conference on [82] V. Haarslev and R. Möller. Racer: A Core Inference En- Data Mining Workshops, pages 625–630, Washington, DC, gine for the Semantic Web. In Proceedings of the 2nd USA, 2007. IEEE Computer Society. International Workshop on Evaluation of Ontology-based [98] K. Janowicz and M. Compton. The stimulus-sensor- Tools (EON2003), located at the 2nd International Semantic observation ontology design pattern and its integration into Web Conference, Sanibel Island, Florida, USA, pages 27–36, the semantic sensor network ontology. In Proceedings of 2003. the 3rd International workshop on Semantic Sensor Networks [83] J. K. Hart and K. Martinez. Environmental sensor networks:a 2010 (SSN10) in conjunction with the 9th International Se- revolution in the earth system science? Earth-Science Re- mantic Web Conference (ISWC 2010), Shanghai, China, Sept. views, 78:177–191, 2006. 2010. [84] K. Henricksen and J. Indulska. Modelling and using imper- [99] Apache Jena. http://jena.apache.org/. fect context information. In PerCom Workshops, pages 33– [100] C. Jiang and P. Steenkiste. A hybrid location model with a 37, 2004. computable location identifier for ubiquitous computing. In [85] M. Hepp. Goodrelations: An ontology for describing products Proceedings of UbiComp ’02, pages 246–263, Gothenberg, and services offers on the web. In Proceedings of the 16th in- Sweden, 2002. Springer-Verlag. ternational conference on Knowledge Engineering: Practice [101] L. Kagal, T. Finin, and A. Joshi. A policy based approach to and Patterns, EKAW ’08, pages 329–346, Berlin, Heidelberg, security for the semantic web. In Proceedings of the second 2008. Springer-Verlag. International Semantic Web Conference (ISWC 2003), pages [86] M. Hert, G. Reif, and H. C. Gall. A comparison of RDB- 402–418, Sanibel Island, Florida, USA, 2003. to-RDF mapping languages. In Proceedings of the 7th Inter- [102] L. Kagal, V. Korolev, H. Chen, A. Joshi, and T. Finin. Centau- national Conference on Semantic Systems, I-Semantics ’11, rus: A framework for intelligent services in a mobile environ- pages 25–32, Graz, Austria, 2011. ACM. ment. In ICDCSW ’01: Proceedings of the 21st International [87] J. Hightower, B. Brumitt, and G. Borriello. The location Conference on Distributed Computing Systems, pages 195– stack: A layered model for location in ubiquitous computing. 201, Washington, DC, USA, April 2001. IEEE Computer So- Proceedings of WMCSA 2002, 00:22–28, 2002. ciety. [88] J. R. Hobbs and F. Pan. An ontology of time for the seman- [103] B. M. Kiernan, S. Beirne, C. Fay, and D. Diamond. Mon- tic web. ACM Transactions on Asian Language Information itoring of gas emissions at landfill sites using autonomous Processing (TALIP), 3(1):66–85, 2004. gas sensors. Technical report, Clarity Centre, Ireland, 2010. [89] F. Hohl, U. Kubach, A. Leonhardi, K. Rothermel, and Project Report. STRIVE, Environmental Protection Agency. M. Schwehm. Next century challenges: Nexus—an open [104] M. Kifer, G. Lausen, and J. Wu. Logical foundations of global infrastructure for spatial-aware applications. In Mo- object-oriented and frame-based languages. Journal of ACM, Ye et alia / Semantically-enriched pervasive sensor-driven systems 41

42(4):741–843, 1995. [121] R. E. McGrath, A. Ranganathan, R. H. Campbell, and M. D. [105] S. Klarman, U. Endriss, and S. Schlobach. ABox Abduc- Mickunas. Use of Ontologies in a Pervasive Computing Envi- tion in the Description Logic ALC. J. Autom. Reasoning, ronment. Technical Report UIUCDCS-R-2003-2332 UILU- 46(1):43–80, 2011. ENG-2003-1719, Department of Computer Science, Univer- [106] M. Klusch. Chapter 4: Semantic web service coordination. In sity of Illinois, Urbana-Champaign, Urbana, Illinois, 2003. CASCOM - Intelligent Service Coordination in the Semantic [122] R. E. McGrath, A. Ranganathan, M. D. Mickunas, and R. H. Web. Birkhaeuser Verlag, Springer, 2008. Campbell. Investigations of semantic interoperability in [107] R. A. Kowalski and M. J. Sergot. A logic-based calculus of ubiquitous computing environments. In Proceedings of the events. New Generation Comput., 4(1):67–95, 1986. 15th IASTED International Conference on Parallel And Dis- [108] A. A. Krisnadhi, K. Sengupta, and P. Hitzler. Local closed tributed Computing And Systems (PDCS 2003), Marina del world semantics: Keep it simple, stupid! In Description Log- Rey, CA, USA, November 2003. ics, 2011. [123] S. McKeever, J. Ye, L. Coyle, and S. Dobson. A multilayered [109] K. N. Kumar, R.Prabakaran, V. S. Dhulipala, and P.Ranjith. uncertainty model for context aware systems. In Adjunct Pro- Future sensors and utilization of sensors in chemical indus- ceedings of the international conference on Pervasive Com- tries with control of environmental hazards. In Proceedings of puting: Late Breaking Result, pages 1–4, Sydney, Australia, the 2nd International Conference on Environmental Science May 2008. and Development, pages 224–228. IACSIT Press, 2011. [124] G. Meditskos and N. Bassiliades. A Rule-Based Object- [110] U. Küster, H. Lausen, and B. König-Ries. Evaluation of se- Oriented OWL Reasoner. IEEE Trans. Knowl. Data Eng., mantic service discovery – a survey and directions for fu- 20(3):397–410, 2008. ture research. In Post-Proceedings of the 2nd Workshop on [125] G. Meditskos and N. Bassiliades. DLEJena: A practical Emerging Web Services Technology (WEWST07) in conjunc- forward-chaining OWL 2 RL reasoner combining Jena and tion with the 5th IEEE European Conference on Web Services Pellet. Web Semant., 8(1):89–94, Mar. 2010. (ECOWS07), 2007. [126] S. B. Mokhtar, D. Preuveneers, N. Georgantas, V. Issarny, and [111] D. B. Lenat and R. V.Guha. Building Large Knowledge-Based Y. Berbers. Easy: Efficient semantic service discovery in per- Systems; Representation and Inference in the Cyc Project. vasive computing environments with qos and context support. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, Journal of Systems and Software, 81(5):785 – 808, 2008. USA, 1989. [127] L. Moreau, B. Clifford, J. Freire, J. Futrelle, Y. Gil, P. Groth, [112] L. Li and I. Horrocks. A software framework for matchmak- N. Kwasnikowska, S. Miles, P. Missier, J. Myers, B. Plale, ing based on semantic web technology. In Proceedings of Y. Simmhan, E. Stephan, and J. V. den Bussche. The Open the 12th international conference on World Wide Web, WWW Provenance Model core specification (v1.1). Future Genera- ’03, pages 331–339, Budapest, Hungary, 2003. ACM. tion Computer Systems, July 2010. [113] P. Lilis, I. Lourdi, C. Papatheodorou, and M. Gergatsoulis. A [128] T. Moser, H. Roth, S. Rozsnyai, R. Mordinyi, and S. Biffl. Se- metadata model for representing time-dependent information mantic event correlation using ontologies. In Proceedings of in cultural collections. In Proceedings of the first online meta- the Confederated International Conferences, CoopIS, DOA, data and semantics research conference (MTSR’ 05), pages IS, and ODBASE 2009 on On the Move to Meaningful Inter- 1–12. Rinton Press, Nov. 2005. net Systems: Part II, OTM ’09, pages 1087–1094, Vilamoura, [114] B. Logan, J. Healey, M. Philipose, E. M. Tapia, and S. S. In- Portugal, 2009. Springer-Verlag. tille. A long-term evaluation of sensing modalities for activity [129] B. Motik, I. Horrocks, R. Rosati, and U. Sattler. Can OWL recognition. In UbiComp’07: Proceedings of the 9th interna- and logic programming live together happily ever after? In In- tional conference on Ubiquitous computing, pages 483–500, ternational Semantic Web Conference, pages 501–514, 2006. Innsbruck, Austria, September 2007. Springer-Verlag. [130] B. Motik and R. Rosati. Reconciling description logics and [115] D. Martin, M. Burstein, and J. Hobbs. rules. J. ACM, 57(5):30:1–30:62, June 2008. OWL-S: Semantic markup for web services. [131] B. Motik, U. Sattler, and R. Studer. Query Answering for http://www.w3.org/Submission/OWL-S, 2004. OWL-DL with rules. J. Web Sem., 3(1):41–60, 2005. [116] K. Martinez, J. K. Hart, and R. Ong. Environmental sensor [132] B. Motik, R. Shearer, and I. Horrocks. Hypertableau reason- networks. Computer, 37(8):50–56, Aug. 2004. ing for description logics. J. Artif. Int. Res., 36(1):165–228, [117] B. McBride. Jena: A semantic web toolkit. IEEE Internet Sept. 2009. Computing, 6(6):55–59, 2002. [133] M. Mühlenbrock, O. Brdiczka, D. Snowdon, and J.-L. Meu- [118] B. McBride and M. Butler. Representing and querying his- nier. Learning to detect user activity and availability from a torical information in RDF with application to E-discovery. variety of sensor data. In PERCOM ’04: Proceedings of the Technical Report HPL-2009-261, Hewlit Packard Laborato- Second IEEE International Conference on Pervasive Comput- ries, 2009. ing and Communications, pages 13 – 22, Orlando, Florida, [119] I. McCowan, D. Gatica-Perez, S. Bengio, G. Lathoud, Mar. 2004. IEEE Computer Society. M. Barnard, and D. Zhang. Automatic analysis of multimodal [134] The NeOn Toolkit. http://neon-toolkit.org. group actions in meetings. IEEE Transactions on Pattern [135] R. Nevatia, J. Hobbs, and B. Bolles. An ontology for video Analysis and Machine Intelligence, 27:305–317, 2005. event representation. In Proceedings of the 2004 Confer- [120] R. E. McGrath. Discovery and Its Discontents: Discov- ence on Computer Vision and Pattern Recognition Workshop ery Protocols for Ubiquitous Computing. Technical Re- (CVPRW’04) Volume 7, CVPRW ’04, pages 119–129, Wash- port UIUCDCS-R-99-2132, Department of Computer Sci- ington, DC, USA, 2004. IEEE Computer Society. ence University of Illinois Urbana-Champaign , Urbana, [136] N. F. Noy, R. W. Fergerson, and M. A. Musen. The knowl- March 2000. edge model of Protége-2000: Combining interoperability and 42 Ye et alia / Semantically-enriched pervasive sensor-driven systems

flexibility. In Proceedings of the 2th International Confer- ence on World Wide Web (WWW ’08), pages 605–614, Bei- ence on Knowledge Engineering and Knowledge Manage- jing, China, 2008. ACM. ment (EKAW’ 2000), pages 17–32, Juan-les-Pins, France, [154] A. Ranganathan, J. Al-Muhtadi, and R. H. Campbel. Reason- 2000. Springer. ing about uncertain contexts in pervasive computing environ- [137] M. J. O’Connor and A. K. Das. SQWRL: A query lan- ments. IEEE Pervasive Computing, 3(2):62–70, 2004. guage for OWL. In OWL: Experiences and Directions [155] A. Ranganathan, J. Al-Muhtadi, S. Chetan, R. Campbell, and (OWLED), volume 529 of CEUR Workshop Proceedings. M. D. Mickunas. Middlewhere: a middleware for location CEUR-WS.org, 2009. awareness in ubiquitous computing applications. In Proceed- [138] M. J. O’Connor and A. K. Das. A lightweight model ings of Middleware ’04, pages 397–416, New York, USA, for representing and reasoning with temporal information in 2004. Springer-Verlag New York, Inc. biomedical ontologies. In Proceedings of the International [156] A. Ranganathan and R. H. Campbell. An infrastructure for Conference on Health Informatics(HEALTHINF 2010), Va- context-awareness based on first order logic. Personal and lencia, Spain, 2010. Ubiquitous Computing, 7:353–364, 2003. [139] F. Orlandi and A. Passant. Modelling provenance of dbpe- [157] A. Ranganathan, R. E. McGrath, R. H. Campbell, and M. D. dia resources using wikipedia contributions. J. Web Sem., Mickunas. Use of Ontologies in a Pervasive Computing En- 9(2):149–164, 2011. vironment. Knowledge Engineering Review, 18(3):209–220, [140] F. Paganelli and D. Giuli. An ontology-based system 2004. for context-aware and configurable services to support [158] A. S. Rao and M. P. Georgeff. BDI agents: From theory to home-based continuous care. Information Technology in practice. In ICMAS, pages 312–319. The MIT Press, 1995. Biomedicine, IEEE Transactions on, 15(2):324 –333, March [159] D. Riboni and C. Bettini. Cosar: hybrid reasoning for context- 2011. aware activity recognition. Personal and Ubiquitous Comput- [141] F. Pan and J. R. Hobbs. Time ontology in ing, 15(3):271–289, 2011. OWL. W3C working draft, W3C, Sept. 2006. [160] D. Riboni and C. Bettini. OWL 2 modeling and reasoning http://www.w3.org/TR/2006/WD-owl-time-20060927/. with complex human activities. Pervasive and Mobile Com- [142] J. Paradiso, P. Dutta, H. Gellersen, and E. Schooler. Guest puting, 7(3):379–395, 2011. editors’ introduction: Smart energy systems. Pervasive Com- [161] M. Roman, C. K. Hess, R. Cerqueira, A. Ranganathan, R. H. puting, IEEE, 10(1):11 –12, Jan-Mar 2011. Campbell, and K. Nahrstedt. Gaia: A Middleware Infrastruc- [143] A. Paschke, H. Boley, and P. Vincent. Semantic complex ture to Enable Active Spaces. IEEE Pervasive Computing, event processing – the future of dynamic IT. Invited talk at pages 74–83, Oct–Dec 2002. SemTech 2010, June 2010. [162] R. Rosati. DL+log: Tight integration of description logics [144] P. F. Patel-Schneider and I. Horrocks. A comparison of two and disjunctive datalog. In P. Doherty, J. Mylopoulos, and modelling paradigms in the semantic web. J. Web Sem., C. A. Welty, editors, Proceedings of the tenth International 5(4):240–250, 2007. Conference on Principles of Knowledge Representation and [145] T. Patkos, I. Chrysakis, A. Bikakis, D. Plexousakis, and Reasoning, pages 68–78. AAAI Press, 2006. G. Antoniou. A reasoning framework for ambient intel- [163] S. Rozsnyai, R. Vecera, J. Schiefer, and A. Schatten. Event ligence. In Proceedings of the 6th Hellenic conference cloud - searching for correlated business events. In IEEE In- on Artificial Intelligence: theories, models and applications, ternational Conference on E-Commerce Technology and En- SETN’10, pages 213–222, Athens, Greece, 2010. Springer- terprise Computing, E-Commerce, and E-Services, volume 0, Verlag. pages 409–420, Los Alamitos, CA, USA, 2007. IEEE Com- [146] I. S. E. Peraldi, A. Kaya, and R. Möller. Formalizing multime- puter Society. dia interpretation based on abduction over description logic [164] D. Russomanno, C. Kothari, and O. Thomas. Sensor ontolo- ABoxes. In Description Logics, volume 477 of CEUR Work- gies: From shallow to deep models. In Proceedings of the shop Proceedings, 2009. 37th Southeastern Symposium on System Theory (SSST ’05), [147] F. Perich, A. Joshi, T. Finin, and Y. Yesha. On data manage- pages 107–112, Mar. 2005. ment in pervasive computing environments. IEEE Trans. on [165] D. J. Russomanno, C. R. Kothari, and O. A. Thomas. Build- Knowl. and Data Eng., 16(5):621–634, May 2004. ing a sensor ontology: A practical approach leveraging ISO [148] M. Perry, A. Sheth, and P. Jain. SPARQLST:Extending and OGC models. In Proceedings of the 2005 International SPARQL to Support Spatiotemporal Queries. Technical Re- Conference on Artificial Intelligence, pages 637–643, Las Ve- port KNOESIS-TR-2009-01, Kno.e.sis Center Technical Re- gas, USA, 2005. port, 2009. [166] K. Sagonas, T. Swift, and D. S. Warren. Xsb as an efficient de- [149] PROV-O: The PROV Ontology, W3C Working Draft 03 May ductive database engine. In In Proceedings of the ACM SIG- 2012. http://www.w3.org/TR/prov-o/. MOD International Conferenceon the Management of Data, [150] Provenance Vocabulary Core Ontology Specification, 14 pages 442–453. ACM Press, 1994. March 2012. http://trdf.sourceforge.net/ [167] S. Santini, B. Ostermaier, and A. Vitaletti. First experiences provenance/ns.html. using wireless sensor networks for noise pollution monitor- [151] Provenance Working Group. http://www.w3.org/ ing. In Proceedings of the workshop on Real-world wire- 2011/prov/wiki/Main_Page. less sensor networks, REALWSN ’08, pages 61–65, Glasgow, [152] Provenir Ontology. http://wiki.knoesis.org/ Scotland, 2008. ACM. index.php/Provenir_Ontology. [168] A. Scherp, T. Franz, C. Saathoff, and S. Staab. A core ontol- [153] A. Pugliese, O. Udrea, and V. S. Subrahmanian. Scaling RDF ogy on events for representing occurrences in the real world. with Time. In Proceeding of the 17th international confer- Multimedia Tools and Applications, 58(2):293–331, 2012. Ye et alia / Semantically-enriched pervasive sensor-driven systems 43

[169] A. Scherp, S. Papadopoulos, A. Kritikos, F. Schwa- [184] T. Strang and C. Linnhoff-Popien. Service interoperabil- gereit, C. Saathoff, T. Franz, D. Schmeiss, S. Staab, ity on context level in ubiquitous computing environments. S. Schenk, and M. Bonifacio. Prototypical Knowl- In Proceedings of International Conference on Advances in edge Management Methodology (Deliverable D5.2.1), We- Infrastructure for Electronic Business, Education, Science, KnowIt Project. http://www.weknowit.eu/sites/ Medicine, and Mobile Technologies on the Internet (SS- default/files/D5.2.1.pdf, 2009. GRR2003w), L’Aquila/Italy, January 2003. [170] H. R. Schmidtke and W. Woo. Towards ontology-based for- [185] T. Strang and C. Linnhoff-Popien. A context modeling sur- mal verification methods for context aware systems. In Per- vey. In Proceedings of the Workshop on Advanced Con- vasive ’09: Proceedings of the 7th international conference text Modelling, Reasoning and Management as part of Ubi- on pervasive computing, pages 309–326, Nara, Japan, 2009. Comp 2004 - The Sixth International Conference on Ubiqui- Springer-Verlag. tous Computing, Nottingham/England, September 2004. [171] openRDF.org. http://www.openrdf.org/. [186] T. Strang, C. Linnhoff-Popien, and K. Frank. Applications of [172] M. Shanahan. Perception as abduction: Turning sensor a Context Ontology Language. In D. Begusic and N. Rozic, data into meaningful representation. Cognitive Science, editors, Proceedings of the International Conference on Soft- 29(1):103–134, 2005. ware, Telecommunications and Computer Networks (Soft- [173] R. Shaw, R. Troncy, and L. Hardman. Lode: Linking open Com2003), pages 14–18. Faculty of Electrical Engineering, descriptions of events. In Proceedings of the 4th Asian Con- Mechanical Engineering and Naval Architecture, University ference on The Semantic Web, ASWC ’09, pages 153–167, of Split, Croatia, October 2003. Shanghai, China, 2009. Springer-Verlag. [187] T. Strang, C. Linnhoff-Popien, and K. Frank. CoOL: A Con- [174] E. Sirin, B. Parsia, B. C. Grau, A. Kalyanpur, and Y. Katz. text Ontology Language to enable Contextual Interoperabil- Pellet: A practical OWL-DL reasoner. Web Semantics: Sci- ity. In J.-B. Stefani, I. Dameure, and D. Hagimont, edi- ence, Services and Agents on the World Wide Web, 5(2):51– tors, LNCS 2893: Proceedings of 4th IFIP WG 6.1 Interna- 53, 2011. tional Conference on Distributed Applications and Interop- [175] S. Song, M. Kim, S. Rho, and E. Hwang. Music ontology for erable Systems (DAIS2003), Lecture Notes in Computer Sci- mood and situation reasoning to support music retrieval and ence (LNCS), pages 236–247, Paris/France, November 2003. recommendation. In Proceedings of the 2009 Third Interna- Springer Verlag. tional Conference on Digital Society, ICDS ’09, pages 304– [188] N. Streitz and P. Nixon. Introduction. Commun. ACM, 309, Washington, DC, USA, 2009. IEEE Computer Society. 48(3):32–35, Mar. 2005. [176] T. Springer and A.-Y. Turhan. Employing description log- [189] IEEE Suggested Upper Merged Ontology. http://suo. ics in ambient intelligence for modeling and reasoning about ieee.org/SUO/SUMO/SUMO_173.kif. complex situations. JAISE, 1(3):235–259, 2009. [190] J. Tao, E. Sirin, J. Bao, and D. L. McGuinness. Integrity Con- [177] G. Stevenson, S. Knox, S. Dobson, and P. Nixon. Ontonym: straints in OWL. In Proceedings of Twenty-Fourth AAAI Con- a collection of upper ontologies for developing pervasive sys- ference on Artificial, 2010. tems. In Proceedings of the 1st Workshop on Context, Infor- [191] J. Tappolet and A. Bernstein. Applied Temporal RDF: Effi- mation and Ontologies, CIAO ’09, pages 9:1–9:8, Heraklion, cient Temporal Querying of RDF Data with SPARQL. In Pro- Greece, 2009. ACM. ceedings of the 6th European Semantic Web Conference on [178] G. Stevenson, J. Ye, and S. Dobson. On the impact of the The Semantic Web, pages 308–322, Heraklion, Crete, Greece, temporal features of sensed data on the development of per- 2009. Springer-Verlag. vasive systems. In Proceedings of the International Workshop [192] K. Taylor and L. Leidinger. Ontology-driven complex event on Programming Methods for Mobile and Pervasive Systems processing in heterogeneous sensor networks. In Proceedings (PMMPS’ 10), Helsinki, Finland, May 2010. of the 8th extended semantic web conference on The semantic [179] G. Stevenson, J. Ye, S. Dobson, and P. Nixon. Loc8: A loca- web: research and applications - Volume Part II, pages 285– tion model and extensible framework for programming with 299, Heraklion, Crete, Greece, 2011. Springer-Verlag. location. IEEE Pervasive Computing, 9(1):28 – 37, January [193] H. J. ter Horst. Extending the rdfs entailment lemma. In 2010. International Semantic Web Conference, pages 77–91, 2004. [180] G. Stoilos, G. B. Stamou, and J. Z. Pan. Fuzzy extensions [194] H. J. ter Horst. Completeness, decidability and complexity of of owl: Logical properties and reduction to fuzzy description entailment for rdf schema and a semantic extension involving logics. Int. J. Approx. Reasoning, 51(6):656–679, 2010. the owl vocabulary. Web Semant., 3(2-3):79–115, Oct. 2005. [181] G. Stoilos, G. B. Stamou, J. Z. Pan, V. Tzouvaras, and I. Hor- [195] K. Teymourian and A. Paschke. Semantic rule-based complex rocks. Reasoning with very expressive fuzzy description log- event processing. In Proceedings of the 2009 International ics. J. Artif. Intell. Res. (JAIR), 30:273–320, 2007. Symposium on Rule Interchange and Applications, RuleML [182] U. Straccia. Towards a fuzzy description logic for the se- ’09, pages 82–92, Las Vegas, Nevada, 2009. Springer-Verlag. mantic web (preliminary report). In Proceedings of the Sec- [196] Y. Theoharis, I. Fundulaki, G. Karvounarakis, and ond European conference on The Semantic Web: research and V. Christophides. On provenance of queries on semantic web Applications, ESWC’05, pages 167–181, Heraklion, Greece, data. IEEE Internet Computing, 15(1):31–39, Jan. 2011. 2005. Springer-Verlag. [197] A. Toninelli, R. Montanari, L. Kagal, and O. Lassila. A se- [183] U. Straccia. Reasoning web. chapter Managing Uncertainty mantic context-aware access control framework for secure and Vagueness in Description Logics, Logic Programs and collaborations in pervasive computing environments. In Pro- Description Logic Programs, pages 54–103. Springer-Verlag, ceedings of the 5th international conference on The Semantic Berlin, Heidelberg, 2008. Web, ISWC’06, pages 473–486, Athens, GA, 2006. Springer- Verlag. 44 Ye et alia / Semantically-enriched pervasive sensor-driven systems

[198] TopBraid Composer. http://www.topquadrant. [212] M. Weiser. The computer for the 21st century. Scientiﬁc com/products/TB_Composer.html. American, 265(3):94–104, Sept. 1991. [199] D. Trastour, C. Bartolini, and J. Gonzalez-Castillo. A Seman- [213] K. J. Witt, J. Stanley, D. Smithbauer, D. Mandl, A. U. V. Ly, tic Web Approach to Service Description for Matchmaking of and M. Metheny. Enabling sensor webs by utilizing SWAMO Services. Technical Report HPL-2001-183, HP Laboratories for autonomous operations. In Proceedings of the 8th NASA Bristol, 2001. Earth Science Technology Conference (ESTC 2008), 2008. [200] D. Tsarkov and I. Horrocks. FaCT++ Description Logic Rea- [214] T. W. Wlodarczyk, C. Rong, M. O’Connor, and M. Musen. soner: System Description. In International Joint Confer- SWRL-F: a fuzzy logic extension of the semantic web rule ence on Automated Reasoning (IJCAR 2006), pages 292–297, language. In Proceedings of the International Conference on 2006. Web Intelligence, Mining and Semantics, WIMS ’11, pages [201] A.-Y. Turhan, T. Springer, and M. Berger. Pushing doors for 39:1–39:9, Sogndal, Norway, 2011. ACM. modeling contexts with OWL DL a case study. In PerCom [215] J. Ye, L. Coyle, S. Dobson, and P. Nixon. Ontology-based Workshops, pages 13–17, 2006. models in pervasive computing systems. The Knowledge En- [202] M. Vacura, V. Svátek, and P. Smrž. A pattern-based frame- gineering Review, 22(04):315–347, December 2007. work for uncertainty representation in ontologies. In Proceed- [216] J. Ye, L. Coyle, S. Dobson, and P. Nixon. Using situation ings of the 11th international conference on Text, Speech and lattices to model and reason about context. In MRC 2007: Dialogue, TSD ’08, pages 227–234, Brno, Czech Republic, Proceedings of the Workshop on modeling and reasoning on 2008. Springer-Verlag. context (coexist with CONTEXT’07), pages 1–12, Roskilde, [203] W. R. van Hage, V. Malaise, R. H. Segers, L. Hollink, and Denmark, August 2007. G. Schreiber. Design and use of the simple event model (sem). [217] J. Ye, S. Dobson, and S. McKeever. Situation identiﬁcation Web Semantics: Science, Services and Agents on the World techniques in pervasive computing: a review. Pervasive and Wide Web, 9(2), 2011. mobile computing, 8:36–66, Feb. 2012. [204] F. van Harmelen and D. L. McGuinness. OWL Web Ontology [218] J. Ye, S. McKeever, L. Coyle, S. Neely, and S. Dobson. Re- Language Overview. http://www.w3.org/TR/2004/REC-owl- solving uncertainty in context integration and abstraction. In features-20040210/, 2004. ICPS 2008: Proceedings of the international conference on [205] T. van Kasteren, A. Noulas, G. Englebienne, and B. Kröse. Pervasive Services, pages 131–140, Sorrento, Italy, July 2008. Accurate activity recognition in a home setting. In UbiComp ACM. ’08: Proceedings of the 10th International Conference on [219] J. Ye, G. Stevenson, and S. Dobson. A top-level ontology for Ubiquitous Computing, pages 1–9, Seoul, Korea, Sept. 2008. smart environments. Pervasive and mobile computing, 7:359– ACM. 378, June 2011. [206] M. Y. Vardi. Why is modal logic so robustly decidable? In [220] J. Ye, G. Stevenson, S. Dobson, M. O’Grady, and G. O’Hare. Proceedings of a DIMACS Workshop on Descriptive Com- PI: Perceiver and interpreter of smart home datasets. In Pro- plexity and Finite Models, DIMACS Series in Discrete Math- ceedings of the 5th International ICST Conference on Per- ematics and Theoretical Computer Science, pages 149–184. vasive Computing Technologies for Healthcare (Pervasive- American Mathematical Society, Jan. 1996. Health 2011), pages 131–138, Dublin, Ireland, 2011. [207] M. Völkel and Y. Sure. RDFReactor - From Ontologies to [221] Z. Yu and Y. Nakamura. Smart meeting systems: A sur- Programmatic Data Access. In Poster and Demo proceedings vey of state-of-the-art and open issues. ACM Comput. Surv., fo the International Semantic Web Conference (ISWC) 2005, 42(2):8:1–8:20, Mar. 2010. Galway, Ireland, Nov. 2005. [222] Z. Yu, X. Zhou, Z. Yu, J. H. Park, and J. Ma. iMuseum: A [208] W3C Semantic Sensor Network Incubator group. http:// scalable context-aware intelligent museum system. Computer www.w3.org/2005/Incubator/ssn/. Communications, 31(18):4376–4382, 2008. [209] Review of sensor and observation ontologies. [223] J. Yuan, Y. Zheng, X. Xie, and G. Sun. Driving with knowl- http://www.w3.org/2005/Incubator/ssn/ edge from the physical world. In Proceedings of KDD’ 11, wiki/Review_of_Sensor_and_Observations_ pages 316–325, San Diego, California, USA, 2011. Ontologies. [224] W. Zhang, K. M. Hansen, and T. Kunz. Enhancing intelli- [210] X. Wang, J. S. Dong, C.-Y. Chin, S. Hettiarachchi, and gence and dependability of a product line enabled pervasive D. Zhang. Semantic Space: An Infrastructure for Smart middleware. Pervasive and Mobile Computing, 6(2):198– Spaces. IEEE Pervasive Computing, 3(3):32–39, 2004. 217, 2010. [211] X. Wang, D. Zhang, T. Gu, and H. K. Pung. Ontology based [225] Y. Zheng, L. Zhang, X. Xie, and W.-Y. Ma. Mining interest- context modeling and reasoning using owl. In PerCom Work- ing locations and travel sequences from GPS trajectories. In shops, pages 18–22, 2004. Proceedings of the international World Wide Web conference (WWW 2009), pages 791–801, Madrid, Spain, 2009.