Context-Sharing and Mediated Communication for Smart Environments Smart for Communication Mediated and Context-Sharing Communication for Smart Environments

2008:66

DOCTORAL T H E SIS JosefHallberg

Context-sharing and Mediated Context-sharing and Mediated Communication for Smart Environments Smart for Communication Mediated and Context-sharing Communication for Smart Environments

Josef Hallberg

Luleå University of Technology

2008:66 Department of Computer Science and Electrical Engineering Division of Media Technology

Universitetstryckeriet, Luleå 2008:66|: -1544|: - -- 08 ⁄66 -- 

Context-sharing and Mediated Communication for Smart Environments

Josef Hallberg

Department of Computer Science and Electrical Engineering Luleå University of Technology SE–971 87 Luleå Sweden

December 2008

Supervisors Kåre Synnes, Ph.D., Luleå University of Technology Peter Parnes, Associate Professor, Luleå University of Technology Chris Nugent, Professor, University of Ulster, Northern Ireland ii Abstract

The deployment of pervasive computing technology and services has enabled the creation of smart environments which assist people in their daily activities. However, the deployment of technology and services into these smart environments has been obstructed due to many factors, most notably from the lack of proper models, rules and services. This thesis presents novel solutions for context-sharing intended to make utilisation of smart environments less complex and more cost efficient. It also presents solutions for mediated communication services which uses the power of smart environments to improve quality, flexibility and usability. The proposed solutions have all been evaluated through implementation and testing of proof-of-concept prototypes. Smart environments can provide a range of different services, such as giving warnings or reminders. Nevertheless, creation and personalisation of services can be complex and time consuming tasks, which promotes the creation of a common approach for tackling the heterogeneous nature of services and data. This thesis proposes such a common approach in the form of the openHome Suite which handles data collection, data analysis, and decision support for smart environments using XML. HomeML and HomeRuleML, two core components of the openHome Suite, supports sharing of context and rules with other research organizations and developers. It simplifies deployment through the creation of models which can be reused between smart environments. Tools for mediated communication are common in many smart environments. This thesis proposes how to utilise the smart environment to improve multimedia communication. A smart environment enables a user to transfer media content between devices for better quality, cost, and privacy. This thesis proposes context-aware communication and demonstrates how this can be established automatically by using the developed HomeRuleML approach, an approach which enables the representation and exchange of decision support rules. The study of context-aware communication and automatic establishment of communication has resulted in a new concept for communication called "dynamic groups" which is a simple and powerful group communication service. The concept has been proven through the development and evaluation of a prototype for mobile devices using near field communication. The research solutions presented in this thesis have been evaluated in the area of remote healthcare, for which simpler deployment of services into smart environments and better communication possibilities for patients becomes possible. This has the potential knock-on effect of decreased healthcare costs and improved support for the daily activities to improve a patient’s wellbeing. Overall the work is done with an overarching aim for smart environments to help people stay out of hospitals in favour of living a richer life in their own homes.

iii iv Contents

Abstract iii

Preface xi

Publications xiii

Acknowledgments xv

1 Thesis Introduction 1 1.1 Introduction ...... 3 1.2Motivationofwork...... 4 1.2.1 Scenario ...... 5 1.3 Thesis organization ...... 6

2 Thesis Overview 7 2.1 Background ...... 9 2.1.1 Ubiquitous and Pervasive Computing ...... 9 2.1.2 Smart Environments ...... 10 2.1.3 Mediated human-to-human communication ...... 10 2.1.4 Telemedicine and Telecare ...... 11 2.2 Related work ...... 11 2.2.1 Context-models ...... 12 2.2.2 Context-awareness ...... 13 2.2.3 Knowledge- and Context-sharing ...... 15 2.2.4 Rich Communication ...... 16 2.3ResearchQuestions...... 17 2.3.1 Problems Related to Context-sharing ...... 17 2.3.2 Problems Related to Mediated Communication ...... 18

v vi Contents

2.3.3 Problem Related to Healthcare ...... 18 2.4 Delimitations ...... 18 2.5 Research methodology ...... 19 2.6 Roadmap and Overview of included articles ...... 20 2.7 Smart Environments for Remote Healthcare ...... 23 2.7.1 Context-sharing and the openHome Suite ...... 23 2.7.2 Mediated Communication and Dynamic Groups ...... 27 2.7.3 Conclusions ...... 32

3 Remote Healthcare Monitoring and Assessment 33 3.1 Introduction to remote healthcare monitoring ...... 35 3.1.1 Changes in Population Demographics ...... 35 3.1.2 The impact of Technology from social and economic perspectives . . 36 3.1.3 Models of Patient-Healthcare Professional Interaction ...... 37 3.1.4 The impact of the Internet on Healthcare Delivery ...... 38 3.2 Telemedicine ...... 39 3.2.1 What is Telemedicine? ...... 39 3.3 Telecare ...... 41 3.3.1 What is Telecare? ...... 41 3.4AmbientandPervasiveComputing...... 42 3.4.1 Intelligent Homes ...... 42 3.4.2 Wearable Systems and Smart Clothes ...... 43 3.4.3 Data Processing and Context Awareness ...... 43 3.5 Case study: Home Based Medication Management ...... 44 3.5.1 Problem Domain ...... 44 3.5.2 Potential deployment of technology ...... 44 3.5.3 The MEDICATE system ...... 45 3.5.4 Conclusions ...... 47

4 Wearable Systems in Nursing Home Care: Prototyping Experience 49 4.1 Introduction ...... 51 4.2 Scoping the Project ...... 52 4.3PaperPrototyping...... 53 4.3.1 Paper, Pen, and Plastic ...... 53 Contents vii

4.3.2 Paper Prototyping Beneﬁts...... 54 4.4 Moving to Multimodal Devices ...... 55 4.4.1 Wearable Prototype ...... 55 4.4.2 Communication Application ...... 55 4.4.3 Wizard of Oz Testing ...... 56 4.4.4 Feedback From the Nurses ...... 57 4.5FinalRemarks...... 58 4.6Acknowledgments...... 59

5 Supporting Automatic Media Resource Selection Using Context-Awareness 61 5.1 Introduction ...... 63 5.2 Related Work ...... 64 5.3 Media Resource Management ...... 65 5.3.1 Communication Model ...... 65 5.3.2 Media Resource Selection Problems ...... 66 5.4 Automatic Media Resource Selection ...... 68 5.4.1 A Media Resource Selection Algorithm ...... 69 5.4.2 Using MRSA ...... 70 5.5 Evaluation ...... 71 5.5.1 Implementation ...... 71 5.5.2 Interaction between components ...... 72 5.5.3 Scenarios ...... 72 5.6Discussion...... 73 5.6.1 Future work ...... 74 5.6.2 Acknowledgments ...... 75

6 homeML - An Open Standard for the Exchange of Data Within Smart Environments 77 6.1 Introduction ...... 79 6.2 Background ...... 80 6.3 Methods ...... 81 6.4 Evaluation of the Model ...... 83 6.5 Conclusion ...... 84 viii Contents

7 HomeRuleML - A Model for the Exchange of Decision Support Rules Within Smart Environments 87 7.1 Introduction ...... 89 7.2 Background ...... 90 7.3 Methods ...... 92 7.3.1 The Rule Tree ...... 92 7.3.2 The Operation Tree ...... 94 7.4 Evaluation of the model ...... 95 7.4.1 Example scenario ...... 96 7.5Discussion...... 97 7.6Conclusionsandfuturework...... 98 7.7 Acknowledgements ...... 99

8 HomeCI - A visual editor for healthcare professionals in the design of home based care 101 8.1 Introduction ...... 103 8.2 Background ...... 104 8.3 Methods ...... 104 8.3.1 Establishment of scenarios ...... 105 8.3.2 Development of HomeCI user interface ...... 105 8.3.3 Evaluation approach for HomeCI prototype ...... 107 8.4Results...... 109 8.5Discussion...... 109 8.6Conclusions...... 110 8.7 Acknowledgement ...... 110

9 Assessing the Impact of Individual Sensor Reliability Within Smart Living Environments 111 9.1 Introduction ...... 113 9.2 Background ...... 114 9.3 Methods ...... 116 9.3.1 Smart environment ...... 116 9.3.2 Information management ...... 116 9.3.3 Assessment of sensor reliability ...... 118 9.4Results...... 121 9.5Conclusions...... 124 Contents ix

9.6 Acknowledgements ...... 124

10 Creating Dynamic Groups using Context-awareness 125 10.1 Introduction ...... 127 10.1.1 Scenario1-Healthcare ...... 128 10.1.2 Scenario2-Normalevery-day-use ...... 129 10.1.3 Outline of the document ...... 129 10.2 The Dynamic Groups Model ...... 130 10.2.1 Forming and maintaining groups ...... 130 10.3 Implementation ...... 132 10.3.1 Context-platform ...... 132 10.3.2 Rule-engine ...... 133 10.3.3 Proﬁle-system...... 135 10.3.4 Session management in clients ...... 135 10.3.5 Architectural overview ...... 136 10.4 Privacy and security ...... 137 10.5 User evaluation ...... 138 10.6 Discussion ...... 139 10.7 Conclusions and future work ...... 140 10.8 Acknowledgements ...... 140

11 Proﬁle Management for Dynamic Groups 143 11.1 Introduction ...... 145 11.2 Background ...... 146 11.2.1 Dynamic groups ...... 146 11.2.2 Architecture ...... 148 11.3 HomeCom ...... 149 11.3.1 Proﬁle information ...... 150 11.3.2 Privacy ...... 150 11.3.3 Implementation ...... 151 11.4 Evaluation ...... 156 11.4.1 Example scenarios ...... 156 11.4.2 Prototype ...... 158 11.5 Related work ...... 158 11.5.1 XML, OWL, SOUPA ...... 159 x Contents

11.5.2 Instant Messaging ...... 160 11.5.3 Privacy ...... 160 11.6 Discussion ...... 161 11.7 Conclusions and Future Work ...... 161

12 Discussion, Conclusions and Future Work 163 12.1 Discussion ...... 165 12.2 Conclusions ...... 169 12.3 Future Research Directions ...... 170

Bibliography 171

Appendix 185

Appendix A - Abbreviations 187

Appendix B - Glossary 189 Preface

Previous research within the Media Technology Research Group at Luleå University of Tech- nology has been focuses on rich communication environments for collaborative work. For people within this research-group rich communication is a part of everyday life as mediated communication is made possible through the "e-corridor" which is an informal e-meeting 1 session running continuously. This "e-corridor" is different from a normal ofﬁce corridor as people can join it from different geographical locations and can interact with sound, video, chat and a shared electronic whiteboard. In order to create the feeling of a real corridor it is important to create a sense of presence for people joining from different locations. The research presented in this thesis was started because of the perceived need to increase the sense of presence for people present only through virtual means. The use of context awareness was seen as a possible way to provide a greater sense of presence rather than just sound, video, chat, and a shared whiteboard. With context awareness it is possible to provide information about a user’s location, context and situation, such as if a person is busy, or if a user is alone. It could also be a good addition for people who do not wish to send video but still wish to share a lesser grade of presence. Nevertheless, context- aware services have many other uses; they can help improve quality and security for elders living at home, they can help reduce stress at work by providing information when needed, they can assist users in their work and in their free time. It was with this background that the author became involved in several projects with the aim of providing improved and intelligent healthcare services. One of these projects was the Cogknow project [25], a project with the goal of helping people with dementia to have an improved quality of life through provision of a technical solution with the ability to remember, maintain social contact, perform daily life activities and enhance feelings of safety. In order to improve collaboration with the partners involved in the Cogknow project, and to undertake collaborative research, I undertook a visit to the University of Ulster, in Belfast, Northern Ireland. The stay was divided into two visits comprising a total of six months. There I became involved in research to develop an open standard for the representation, storage and exchange of data collected from smart environments. The collaborative research is closely connected with home healthcare and remote health monitoring, and is currently being used as part of the Cogknow project. The research performed at University of Ulster has propagated into the mediated communication research undertaken at Luleå University of Technology.

1The term "e-meeting", or electronic meeting, denotes a group conferencing session which can include video, audio and chat among other media. Rather than requiring a dedicated meeting room, e-meetings can take place from the user’s desktop and can be used for either formal or informal communication.

xi xii Publications

This doctoral thesis consists of an introduction and nine publications. The introduction provides an overview of all papers and their relation to each other. All papers have been, or will be, published at international peer reviewed conferences or journals. I am the main author of three papers and co-author of six papers.

Paper 1 Chris Nugent, Dewar Finlay, Richard Davies, Mark Donnelly, Josef Hallberg, Nor- man Black, David Craig, "Remote Healthcare Monitoring and Assessment",To appear in Journal of Technology and Health Care, 2008. Paper 2 Mikael Drugge, Josef Hallberg, Peter Parnes, and Kåre Synnes, "Wearable Sys- tems in Nursing Home Care: Prototyping Experience",InJournal of IEEE Perva- sive Computing, vol. 5, no. 1, pages 86–91, January–March 2006. Paper 3 Johan Kristiansson, Josef Hallberg, Sara Svensson, Kåre Synnes, Peter Parnes, "Supporting Automatic Media Resource Selection Using Context-Awareness",In Proceedings of the 3:rd International Conference on Advances in Mobile Multimedia (MoMM2005), pages 271–282, Kuala Lumpur, Malaysia, September 2005. Paper 4 Chris Nugent, Dewar Finlay, Richard Davies, Hui Wang, Huiru Zheng, Josef Hall- berg, Kåre Synnes, Maurice Mulvenna, "homeML - An Open Standard for the Ex- change of Data Within Smart Environments",Inthe 5th International Conference On Smart Homes and Health Telematics (ICOST), pages 121–129, Nara, Japan, June 2007. Also published in Lecture Notes in Computer Science, vol. 4541, pages 121– 129, 2007. Paper 5 Josef Hallberg, Chris Nugent, Richard Davies, Kåre Synnes, Mark Donnelly, Dewar Finlay, Maurice Mulvenna, "HomeRuleML - A Model for the Exchange of Decision Support Rules Within Smart Environments",Inthe 3rd Annual IEEE Conference on Automation Science and Engineering (CASE), pages 513–520, Scottsdale, Arizona, USA, September 2007. Paper 6 Chris Nugent, Richard Davies, Josef Hallberg, Mark Donnelly, Kåre Synnes, Michael Poland, Jonathan Wallace, Dewar Finlay, Maurice Mulvenna, David Craig, "HomeCI - A visual editor for healthcare professionals in the design of home based care",Inthe 29th Annual IEEE Conference on Engineering in Medicine and Biology Society (EMBC), pages 2787–2790, Lyon, France, August 2007.

xiii xiv Publications

Paper 7 Chris Nugent, Xin Hong, Josef Hallberg, Dewar Finlay, Kåre Synnes, "Assessing the Impact of Individual Sensor Reliability Within Smart Living Environments", In the 4th Annual IEEE Conference on Automation Science and Engineering (CASE), pages 685–690, Washington DC, USA, August 2008. Paper 8 Josef Hallberg, Mia Backlund-Norberg, Johan Kristiansson, Kåre Synnes, Chris Nugent, "Creating Dynamic Groups using Context-awareness",Inthe 6th Interna- tional ACM Conference on Mobile and Ubiquitous Multimedia (MUM), pages 42–49, Oulu, Finland, December 2007. Paper 9 Josef Hallberg, Mia Backlund-Norberg, Kåre Synnes, Chris Nugent, "Proﬁle Man- agement in Dynamic Groups", Conditionally accepted to the special Springer volume on Intelligent Patient Management, 2009.

The following publications were intentionally left out from the thesis, either because results have been superseded or made redundant by more recent ﬁndings included herein, or because their focus does not lie entirely within the scope of this thesis.

• Marcus Nilsson, Josef Hallberg, Kåre Synnes, "Positioning with Bluetooth",Inthe 10th International Conference on Telecommunications, 2003 • Josef Hallberg, Sara Svensson, Åke Östmark, Per Lindgren, Kåre Synnes, Jerker Dels- ing, "Cross-country Skiers Go On-line", Technical report 2006:04, ISSN 1402-1536, ISRN LTU-TR–06/04–SE, Luleå, Luleå University of Technology, 2006. Extended version of "Enriched Media-Experience of Sport Events",InProceedings of Workshop on Multimedia Computing Systems and Applications (WMCSA 2004), 2004. • Mikael Drugge, Josef Hallberg, Kåre Synnes, Peter Parnes, "Relieving the medical workers’ daily work through wearable and pervasive computing", In the proceedings of the ﬁrst AMI Work Communities Forum Day 2005: Towards ambient intelligence at work, vision 2010, 2005

• Anita Melander-Wikman, Maria Jansson, Josef Hallberg, Christina Mörtberg, Gunvor Gard, "The lighthouse alarm and locator trial: a pilot study",InJournal for Tech- nology and Healthcare, 2007 • Chris Nugent, Richard Davies, Mark Donnely, Josef Hallberg, Mossaab Hariz, "The development of personalised cognitive prosthetics",Inthe 30th Annual Interna- tional IEEE conference on Engineering in Medicine and Biology Society (EMBC), 2008 Acknowledgments

First I would like to extend my eternal gratitude to my Lord and Saviour for getting me to where I am today. I would also like to thank my family who have always been there to support me, for better and for worse. I would not have gotten far in life without being able to come home to a loving family every now and then. Special thanks also go to my friends. Thank you for all your support and for helping me maintain a life outside of work. The work presented in this thesis could not have been done without the support and en- couragement from my supervisors Kåre Synnes, Peter Parnes and Chris Nugent who have helped me develop ideas and further my research. I thank you for all your valuable insights and comments. I would like to thank my colleges within the Media Technology research group who have always made me feel appreciated. Thank you for taking part in my research and letting me take part in yours. I would also like to thank all the people I have worked with at University of Ulster. Thank you for welcoming me as a guest researcher. Being part of your research on smart environments provided me with valuable insights in the ﬁeld and research in general. Big thanks go to the people at the Centre for Distance-Spanning Healthcare (CDH) and the Centre for Distance-Spanning Technology (CDT), it has been great working with you. Thanks also go to Susanne Andersson (Northern County Council) and Anna-Lena Andersson (Luleå municipality) for providing valuable insights and help with testing and evaluations of prototypes, Johan Kristiansson and Stefan Håkansson at Ericsson Research, Luleå, for your involvement in my research, and Arkady Zaslavsky for your help with this thesis. Finally I would like to thank all the people I have had the pleasure of working with in different projects. I look forward to working with you all again and hope we will accomplish great things together. A special thanks also goes to all my co-authors. It has been a pleasure working with you, and I would never have accomplished this work without you. My research has primarily been funded by CDH and CDT through projects at European, national and regional levels, such as Alipes, Cogknow, eFamily, MemoryLane, MobiGroup, MobiHealth, MyMates, Situate, and SIV.

Luleå, November 2008 Josef Hallberg

xv xvi Part 1

Thesis Introduction

Thesis Introduction 3

1.1 Introduction

We live in an age where computer technology is becoming a natural part of our everyday lives. Computers help us maintain the quality of life we set for ourselves by simplifying communication, providing information access and helping us organize activities. This is made possible by the rapid development of computer technology, where cost and size of new devices are constantly being reduced while battery capacity and mobility of these devices are being increased. Nevertheless, there is only so much we can do with devices we carry with us. Many context-aware services rely on information from devices in our surrounding environment. This introduces the need for smart environments, which can collect sensor data and interact with devices embedded in it. To make sense of collected sensor data in a smart environment there is a need for models and rules. Nevertheless, such models and rules would have to be tailored to a speciﬁc service, person, and environment to work efﬁciently, which can be very time consuming. Knowledge sharing has long been a method to speed up the development of models [89] and can be applied to models for smart environments as well. Sharing knowledge, or contexts, from smart environments between developers and research organizations can speed up development and increase reuse of necessary models and rules, which may reduce complexity and cost of deploying technology and services into smart environments while improving their reliability. This thesis presents models for context- and rule-sharing for this purpose. As previously mentioned the devices we carry with us, such as mobile phones and personal digital assistants, have limitations. This is true for communication as well. While these devices may provide both audio and video communication, the limited size causes limitations in picture quality, usability, and input capabilities. These limitations can be potentially overcome by utilising smart environments which provide access to sensors and devices in the surrounding environment. This thesis presents new concepts for mediated communication, among those a model for group communication. These concepts utilises smart environments to achieve new functionality, and to overcome some of the limitations witnessed in traditional communication devices (e.g. mobile phones). The research presented in this thesis has resulted in the development of several proof-of- concept prototypes. Among these is a smart environment with the purpose of determining behavioural patterns and recognizing daily activities. It stores information using an implementation of the context sharing model for which data can be viewed and analyzed through the complementary browser. Another prototype is a mobile implementation of the group communication model, which demonstrates how group management and group communication can be made simple, even from a mobile-phone. Both prototypes, together with the models they are based on, have been developed further in both national and international projects. Evaluation of these prototypes is being performed with real users and possible further deployment is also being planned in collaboration with industry. This doctoral thesis presents research on how to simplify the development of models and rules for smart environments through context sharing, and how to improve mediated communication using the capabilities of smart environments. The overall research vision is to create models, tools, and services which reduce complexity and cost of deploying technology and services into smart environments while enabling new context-aware services, such as 4 Thesis Introduction daily activity support and context-aware communication. The goal of the presented research is not to provide a complete solution, but rather to present building blocks which can be used in the future to improve applications or services deployed within smart environments.

1.2 Motivation of work

An area which can greatly benefit from the utilisation of smart environments is healthcare. One of the major health issues we face today are the illnesses associated with ageing. At a global level, the population size is increasing, and with this is the percentage of those aged 65 and over [2]. From a healthcare perspective this offers a number of challenges. First of all, we are faced by the simple fact that the population is now larger and hence places a larger demand on healthcare services. Secondly, with an increasing percentage of those aged 65 and over there is an increased prevalence of healthcare problems and instances of disease and thirdly there are now fewer younger people to care for the elderly [115]. It is clear that there is a need for a paradigm shift in healthcare provision when considering the facts as stated above, as meeting the needs from a growing and ageing population becomes increasingly more difficult for healthcare organizations operating on an already strained econ- omy. While today much of the healthcare is bound to a physical location, it could become a decentralized service which caters for the needs of people in their homes, which makes it possible for a patient to receive specialist care or to choose a specific carer. Furthermore, technology provision has moved from reactive care to preventive care, which also supports the need for a decentralized service as preventive measurements aims at preventing institutionalisation. A decentralized service could also help ease collaboration between healthcare personnel which are unable to meet face-to-face at the time. As an end-result this may both increase efficiency in healthcare services, as well as improve the quality of life for patients. The research presented in this thesis approaches this new paradigm from two different directions, firstly through the utilisation of smart environments, and secondly through mediated human-to-human communication. Both are important components for decentralized healthcare. Smart environments make it possible to support everyday activities, while assisting the user and carers to understand behaviour over a longer period of time. Mediated human-to-human communication helps users maintain their social network, but is also a key component in making it possible for healthcare personnel and relatives to keep in touch with the patient. Furthermore, it is also useful for maintaining an efficient information exchange between carers. The research in this thesis has been applied to healthcare environments and the overall theme has been to address some of the primary issues involved in moving towards decentralized healthcare services, namely addressing issues of deployment complexity and maintaining the feeling of closeness to medical personnel as well as friends and family. Section 1.2.1 provides a short scenario which illustrates the envisioned utilisation of results provided in this thesis. Thesis Introduction 5

1.2.1 Scenario

Hulda is 77 years old and is still living at home. This is made possible by the recent deployment of technology and the creation of smart environments in elderly person’s homes, where pervasive computer technology is a central part. This is to help monitor Hulda’s wellbeing, but also to assist her in everyday tasks. Hulda and a carer have together selected a number of suitable services which are governed by a set of defined rules, all from an exemplar list, which are deemed as being suitable to help her and make her life a little easier. In the morning when Hulda wakes up she is reminded to take her medicine. The system clears the reminder after noticing that today’s pill container has been emptied. Hulda always start her day with a cup of tea in her living room. When she sits down in the sofa in the living room and turns on the TV she is automatically connected to an interactive "breakfast group" where she and her friends use to have a morning chat together whilst having their breakfast. After Hulda has had her cup of tea she prepares for her morning exercise. Her smart clothing is equipped with a number of health monitoring sensors, which can measure her ECG among other things. This morning Hulda becomes more exhausted than normal, and this is detected by the system which automatically contacts the local carer who can help her. Due to the possibility of an emergency the camera closest to Hulda is turned on to help the carer assess the severity of the situation. In this scenario the user selects the rules for each service from a list. This list is made possible through context sharing, where others have spent time analyzing data, defining rules, and testing rules, then made the results available for other homes. The scenario illustrates several different rules, as well as the use of dynamic groups which is a solution for flexible mediated human-to-human group communication. The "medication" rule reminds the user to take her pills (e.g. by voice reminder from a speaker or presenting a notice on the TV screen), the "breakfast"-rule connects the user to others having breakfast, and the health monitoring rule monitors the user’s vital signs for anomalies and takes necessary actions in instances of emergency. The rule for health monitoring is made possible through prior analysis of shared health-related data from other users, but was personalized for this particular user. In this scenario the user could also communicate with friends and healthcare personnel. The difference depicted in the scenario compared to a conventional means of mediated communication (i.e. telephone) is the ability to predefine behaviour with rules, to use group communication, and to use devices located in the environment to improve quality of the media used for communicating. Sound and video was transferred to the TV in the living room as the user sat down in the sofa, at the same time the user got an option to show a video of herself through the camera located on the top of the TV. The user could then interact in group with others having breakfast. Finally, when the system noticed an anomaly in vital signs during exercise the system established communication with the carer and selected a nearby camera to help the carer assess the situation. 6 Thesis Introduction

1.3 Thesis organization

The thesis consists of twelve parts. Part 2 presents an overview of this thesis. It provides a presentation of background information relating to the research areas within this Thesis. Fol- lowing this, a number of relevant research questions are presented, followed by a discussion of the research methodology used to address them. Then follows a brief introduction to the papers included in this thesis. The papers are included in this thesis in Part 3 to Part 11, which each contain a paper that has either been published or has been accepted for publication. This thesis is then concluded in Part 12 which contains a discussion, conclusions and future work. The papers are reproduced in original form and have not been modiﬁed since the time of publication, with the following exceptions:

• The formatting of the papers have been unified so that they all share a common style and appearance. • Figures have been resized and repositioned so as to fit aesthetically in the common layout used. • Figures, tables and sections have been renumbered to fit into the numbering scheme used throughout the thesis. • Bibliographical entries and citations have been renumbered, and all references have been moved into a common bibliography at the end of the thesis. • Editorial changes of grammar and spelling have been undertaken to correct a few minor errors. Part 2

Thesis Overview

Thesis Overview 9

2.1 Background

This section presents background information relevant to the research presented in this thesis. Ubiquitous and pervasive computing explains the concepts from which smart environments have sprung. Mediated human-to-human communications is a large area of research, and some of the important terms and concepts will be explained.

2.1.1 Ubiquitous and Pervasive Computing

The terms ubiquitous computing and pervasive computing denote areas of research which are closely related. The general idea for both areas is to bring computational power away from the confinements of a desktop computer and into the real world. The two terms ubiquitous and pervasive are sometimes used interchangeably, but there are some inherent differences in their meaning which should be clarified. The groundwork for the ubiquitous computing area was established in the early 1990’s by Mark Weiser who at that time presented his vision of future computer environments [149]. What he presented was a vision where computers would be available everywhere, and at any time. This research concept was named "ubiquitous computing" and was said to be the beginning of the calm technology age. Today we are beginning to witness the realization of this vision. Computers are now abundant and most people carry one in their pocket in the form of a mobile phone1. In a world so immersed with computers it is important that they do the work for us, rather than adding an extra layer of complexity to our lives. Hence it is desirable to create services which help us with as little interruption to the activities in our everyday life; such a service could be called an "unobtrusive service". Albrecht Schmidt states in his doctoral thesis: "Creating a good experience for a user can improve their perception of their work and so ultimately make the process of using a computer system more effective and pleasurable" [129]. Hence, ubiquitous computing is more than just deploying computers in our environment, it also includes making them user-friendly and useful. Pervasive computing is often used as a synonym to ubiquitous computing, which can be defined as having computers everywhere along with the services they may offer, while hiding the technology from the user. Though there are subtle differences in the definitions of ubiquitous and pervasive computing these concepts are often mixed as it is desirable to provide support everywhere and at any time but without being obtrusive. Pervasive computers should be natural and simple; they should be transparent to the environment. A mobile phone or a smart phone is an example of devices which are commonly used in pervasive computing system environments today. These are small devices we carry close to our person, which offer a natural and common means of providing communication. Mobile technology has been used frequently for creating prototypes of the research presented in this thesis. Smart phones capable of more than just communication can function as both data collection units and as input devices for pervasive services, and have played an important part in the research undertaken by the author.

1There are now over 3 billion mobile subscribers worldwide [16] 10 Thesis Overview

2.1.2 Smart Environments

Smart environments typically involve an infrastructure comprising of sensors for data collection, actuator-based technology which can perform tasks to control the environment, and a number of input and output devices which can facilitate human-computer interaction. Along with this technology there are services which manage recording and processing of data, as well as adapt the environment according to rules. The purposes of these environments can be plentiful, but quite often they are used for monitoring, and for assisting in everyday tasks. Some smart environments are designed to improve the quality of life for the inhabitant. For example, a forgetful person can be reminded by the environment to perform certain tasks, or the environment could simply undertake them on behalf of the user, for example turn off the oven if the user has been absent from the kitchen for a certain period of time. This thesis addresses smart environments that help prolong the time a person can stay at home before being hospitalized or sent to a care centre, as well as to reduce the time it takes until a person can be sent back to the home, has been addressed. Nevertheless, instead of focusing on creating the actual smart environments, this thesis presents possible solutions to some of the current fundamental problems which may help speed up the deployment process and reduce the costs of running services within these environments. Due to their importance of bringing about a new healthcare paradigm, smart environments have been one of the key interest areas of the author.

2.1.3 Mediated human-to-human communication

One term carries extra signiﬁcance when talking about mediated human-to-human communication and it is the term "media richness". This was ﬁrst introduced by Daft and Lengel [30], and refers to the ability of conveying information over time. Richness of communication media can be ranked on their abilities to:

• relay immediate feedback • provide feedback cues such as body language • allow the message to be created or altered speciﬁcally for an intended recipient • transmit the feelings or emotions of the communicators

Face-to-face communication is referred to as the richest form of communication, but rich communication can be achieved without meeting face-to-face. Rich communication can also be achieved through the use of media, also called mediated communication. Mediated human-to-human communication can be achieved using a combination of different media, such as video, audio and chat. The number of different media simply determines the media richness in a conversation. Rich communication is thus mediated communication using more than one media with the aim of immersing the users and thereby increase the perceived quality of the mediated communication. Rich communication has many purposes for healthcare provision. It can be used by someone to maintain a social network, hence avoiding isolation. For example, a family which Thesis Overview 11 normally lives too far away to visit could be present at the dinner table by means of video and audio, and friends could be present in the living room through the television set, even when the weather is bad and the friends prefer not going outside. Rich communication can also be used by healthcare personnel to improve communication. The expert opinion of a doctor could be received despite the doctor being unable to visit the patient, simply by streaming high-resolution or images to the doctor. Mediated human-to-human communication is therefore an important aspect of decentralized healthcare. It is used in the research by the author to provide the means for both contact with and between healthcare personnel and also to maintain a social network. Traditional means for communication (e.g. telephone) is useful, however, the lack of media richness limits its uses. Traditional videoconferencing software can be difﬁcult to use, so combining pervasive technology, smart environments, and rich communication may prove to be an efﬁcient solution to usability problems, and a step closer towards achieving decentralized healthcare provisioning.

2.1.4 Telemedicine and Telecare

The usage of mediated human-to-human communication to provide healthcare may be referred to as telemedicine. Telemedicine can be described as the remote use of medical expertise at the point of need [11]. This results in the use of telecommunications to facilitate the sharing of medical knowledge and the delivery of healthcare over a distance [43]. With this approach medical expertise can be delivered to remote sites, rural areas or understaffed regions. With both ﬁxed and mobile high-speed broadband becoming more readily available it is not unrealistic to expect solutions in the future where a live video-stream captured in the user’s home can provide enough details for a doctor to draw initial conclusion about certain conditions without actually having to visit the patient in person [84]. Another common use of smart environments is through provision of telecare solutions. Telecare combines the usage of sensing devices and telecommunications to support the remote monitoring of a patient within their home environment with the goal of providing a means of care support. A telecare service is designed with the purpose of enabling people to remain independent in their own homes by providing person-centred reactive technologies, and can alert healthcare personnel as well as family in cases of emergencies. It is also possible for the user to raise an alarm manually if support is required, for example if they should fall and require assistance. Quite often smart environments combine telecare with assistive functionality to provide a personalized environment that caters to the user’s needs.

2.2 Related work

The papers included in this thesis contain related work, however this is mainly work related to the research questions addressed by the research presented in each paper respectively. Here however, related work is presented from the perspective of the whole thesis and focuses on context-models, context-awareness, knowledge- and context-sharing, and rich communication. 12 Thesis Overview

2.2.1 Context-models

The term context has had many different definitions, but one that seems generally accepted is the one by Dey and Abowd [1] where context is defined as "any information that can be used to characterize the situation of entities (i.e., whether a person, place or object) that are considered relevant to the interaction between a user and an application, including the user and the application themselves". Modelling context has become increasingly important as the use of context-awareness has increased, but also as context is being used for conveying a form of presence and indication of a user’s activity. The purpose of context modelling is to arrange context data into a format possible to process by computers. There are several different approaches to context-modelling. Strang and Linnhoff-Popient [138] have summarized the most essential approaches which are based on the data structures used for representing and exchanging contextual information in the respective system. These are key-value models, markup scheme models, graphical models, object-oriented models, logic-based models, and ontology-based models. Key-value models build on key-value pairs and are the simplest data structure for modelling contextual information. Early pioneers in the ubiquitous computing area, such as Schilit et al. [125], were using key-value pairs to provide context information (for example location information) to applications using environmental variables. Key-value pairs are often found in various service frameworks used to describe capabilities and attributes of a service. They are simple, just providing attributes and values, but the simplicity also limits its uses. Markup scheme models use a hierarchical structure consisting of markup tags, attributes, and content. Typical uses of the markup scheme models are based on a derivative of Standard Generalized Markup Language, such as the eXtensible Markup Language (XML) [40]. Quite common for markup scheme models is that they focus on a small set of contextual aspects, perhaps an environment or context for a specific service. Examples of uses include the Comprehensive Structured Context Profiles [55], the Pervasive Profile Description Lan- guage [23], and ConteXtML [124]. Graphical models have graphical components which are used to model context, such as the Unified Modelling Language (UML) which was proven useful for modelling context by Bauer [6]. Other uses of graphical modelling of contexts are the Object-Role Modelling approach [56, 57] where entities are associated with different facts (for example is of, permitted to use, located at, engaged in) and categorized according to their persistence (static or dynamic) and source (profiled, sensed, or derived). The graphical model approach is useful to derive an entity-relationship model [21] and to structure information for a relational database in context management systems. Object-oriented models gain the benefits of any object-oriented approach such as being encapsulated and reusable. The common object-oriented approach uses various objects to represent different context types (such as temperature and location) and encapsulates data processing as well as context representation. Access to the information and logical calcula- tions on the context data is provided by well defined interfaces. One of the better known uses of the object-oriented approach is the GUIDE project [22], which utilises an object-oriented model to manage a great variety of personal and environmental contextual information while Thesis Overview 13 maintaining scalability. Logic-based models uses facts, expressions, and rules to model context. Facts are added, updated, and removed through reasoning using the existing rules in the system. Organization of data within logical based models is especially important and there is a need for a high degree of formality. One of the first uses of logic-based models was presented as formalizing context by McCarthy et al. [82] where contexts are introduced as abstract mathematical entities with properties useful in artificial intelligence. Ontology-based models define concepts and relations between concepts. Quite often the management and reasoning of context models are kept separate, but as discussed by Becker and Nicklas in [8] it is possible to combine the two using an ontology-based approach. One of the prominent strengths of ontologies is the way in which contexts from our daily life can be mapped onto a data structures utilised by computers. The CoBrA system [19, 18] is one of the promising approaches utilising ontology-based models, where ontologies are used to characterize entities such as persons, places, and several kinds of entities. The described models have different strengths and uses. Strang and Linhoff-Popient [138] come to the conclusion that ontology-based models are the most expressive and fulfil most of their requirements. This thesis discusses how context and rules can be shared to reduce complexity and cost of deploying smart environments. The logical choice was therefore to use a simple model that could easily be shared in an generally accepted format. Key- value models, markup scheme models, and ontology-based models are all suitable for sharing data, however, key-value models lack a hierarchical structure and ontology-based models add unnecessary complexity thus making them less suitable than markup scheme models. The openHome Suite is therefore utilising the markup scheme, where XML schemas model context and rules. The XML models presented in this thesis are developed for context-sharing and focus on modelling context and rules for smart environments, thus separating them from other markup scheme models.

2.2.2 Context-awareness

Context-awareness means adapting the behaviour of a service or application to the changes in context. The first use of context-awareness was by Schilit and Theimer [127] where it was defined as awareness of location, identities of nearby people, objects, and changes to those objects. There is plenty of research being done on context-awareness but rather than looking at context-aware applications and services this Section will focus on context-aware architectures and frameworks as that is more in line of what is being presented in this thesis. Context-aware architectures and frameworks generally consists of sensing mechanisms, data processing, and context storage using one of the context models described in Section 2.2.1. Some also contain resource discovery mechanisms and solutions for security and privacy. They are commonly often used in smart environments to collect, process, and manage context data which then can be used for sharing contextual information or can be used for context-aware applications and services. The Context Toolkit [34] is perhaps one of the most known architectures for context- aware applications. It consists of 3 main parts: context widgets (distributed sensor units), 14 Thesis Overview context aggregators and context interpreters. The context widgets interface the sensors and hide the underlying context-sensing mechanism(s). The context interpreter abstracts and in- terprets context and transforms data into more convenient forms. Context aggregators collect context from multiple sources to form more complete knowledge about an entity, context or situation. Context Toolkit uses a centralized discoverer where widgets, aggregators, and interpreters will have to be registered in order to be found by clients. The middleware infrastructure Gaia [59, 120] is designed mainly to enable active spaces. Through Gaia, data and services in the environment are always accessible by the user. Per- sonal devices carried by the user are seamlessly integrated with the environment. Gaia is more than just a platform, it is a meta-operating system customized for physical spaces that supports the development of applications customized for these environments. In Gaia there is also advanced data processing which is both rule based and neural-network based [118]. The infrastructure consists of an event manager, a context service, a presence service, a space repository and a context system. The event manager handles various events that may occur within the infrastructure. The context service helps applications and users adapt to changes in context. The presence service keeps updated information about present resources in the environment. The space repository stores information about all software and hardware entities contained in the space. The context system makes sure data is always available and presented in the correct form no matter where and in what situation the user is. BASE [10, 9] is a microbroker-based middleware for pervasive computing. Its purpose is to provide easy-to-use abstractions to remote services and device-specific capabilities. This is to address the problems which arise when the availability of services and resources is constantly changing. A simple signalling mechanism is used to determine the availability of resources and services in the surrounding. While BASE can dynamically select communication protocol stacks it does not offer any support for adaption at higher levels, such as the adaption of services or devices. For this there is the PCOM system [7], a component system for pervasive computing. PCOM uses contracts containing component requirements and the functionality provided by the component. These contracts are then used for service and device selection, and compared to the current needs. In environments with several inhabitants there might be conflicts of interest, and some devices or services might be unsuitable for use. This problem is addressed in COMITY [144], which uses PCOM to handle conflict avoidance in pervasive computing environments. Aura [44, 137] sets out to achieve a platform, or smart environment, for distraction-free ubiquitous computing. That is done by maximising the use of available resources and to minimize both user distraction and drains on user attention. Aura approaches these goals by acting as a proxy for the mobile user it represents: when a user enters a new environment, her Aura marshals the appropriate resources to support her task. The system features location tracking, face recognition, speech recognition and online calendars among other things. With help of these contexts Aura strives to aid users to become effective and productive in their tasks without adding unnecessary distraction or chores. The SOCAM architecture [48, 49] is for building of context-aware mobile services. SO- CAM consists of mainly three parts, the context providers, the context interpreter, and the service locating mechanism. Context providers abstract contexts from different sources and convert them so they can be shared and reused by other SOCAM components. The context Thesis Overview 15 interpreter consists of a context reasoning engine and a context knowledge base, and uses ontologies to represent context data which enable sharing of contextual data. The service locating mechanism enables context providers and the context interpreter to advertise their presence to users or applications. The context broker architecture CoBrA [19, 18] is an agent based architecture for supporting context-aware computing in smart environments. CoBrA manages a shared contextual model on the behalf of a community of agents. These agents can be a variety of applications, services, and objects which interact with their surrounding. The context broker consists of four functional main components: the context knowledge base, the context inference engine, the context acquisition model, and the privacy management module. CoBrA uses COBRA- Ont [18], which is an ontology-based model, to represent its data. The openHome Suite presented in this thesis contains models used for context-sharing. Nevertheless, complementary tools have been developed so that these context-models can be used for storing and processing context. These models were designed specifically for context- sharing and tools for storing and processing context were created in addition to that. Because of this there are differences compared to the architectures presented earlier in this Section. For example, data, which has not been abstracted, is stored in a well structured hierarchical structure to provide a simple overview of data, and to simplify observation of user behaviour. Furthermore the actual model can be populated with data and shared for processing at other research organisations, something which may not be as simple with other architectures. Some architectures offer resource discovery and resource adaption, something which is addressed in "Automatic Media Resource Selection" presented in this thesis (see Section 5). The PCOM system [7] provides device and service adaption, similar to the automatic media resource selection. However, while PCOM utilises a contract with requirements and offered functionality, the work presented in this thesis separates functionality into a number of media sinks and sources, and uses quality, privacy, and costs as metrics for selecting the most suitable resource to use.

2.2.3 Knowledge- and Context-sharing

There are two primary interpretations of context-sharing. One is the sharing of a user’s contexts to others in order to improve their perception of the user’s situation (e.g. sharing location and availability). The other is the sharing of context to improve the common knowledge of a situation and/or problem, and from that devise methods to which this knowledge can be re-applied in new settings. This thesis focus on the latter and presents models for sharing context with the purpose of providing data for analysing, for the creation of decision support rules, and for the creation of services which can be re-applied and reused in different smart environments. It is a fact that knowledge sharing can be beneficial in terms of decision support, methodology, and experience [89] as long as the knowledge can be re-applied in a new setting. Neches et al. write "Building new knowledge-based systems today usually entails construct- ing new knowledge bases from scratch. It could instead be done by assembling reusable components. System developers would then only need to worry about creating the special- 16 Thesis Overview ized knowledge and reasoners new to the specific task of their system. This new system would interoperate with existing systems, using them to perform some of its reasoning. In this way, declaring knowledge, problem-solving techniques, and reasoning services could all be shared among systems. This approach would facilitate building bigger and better systems cheaply. The infrastructure to support such sharing and reuse would lead to greater ubiquity of these systems, potentially transforming the knowledge industry."[91] There are a few approaches to developing models for knowledge-sharing. Pérez and Benjamins write about knowledge sharing and reuse of components using ontologies and problem-solving methods [116]. They state that ontologies define domain knowledge at a generic level, while problem-solving methods specify generic reasoning knowledge. These components can be viewed as complementary and can be used to configure new knowledge systems from existing, reusable components. Another approach using ontologies for knowledge sharing is presented by Gruber [47] where five general design criteria (clarity, coherence, extendibility, minimal encoding bias, and minimal ontological commitment) for the design of formal ontologies for knowledge sharing is given. Another approach for context sharing is DJess [17], which is a lightweight middleware for contextual knowledge (facts) and reactive behaviour (rules) that can be shared between different nodes of a pervasive computing system. Inference systems (agents) communicate and coordinate with each other through the ability to reason on facts asynchronously, which is enabled by the fact that DJess is constructed using Java and uses remote method invocation (RMI) for shared memory between agents. Furthermore, rules can be loaded into a remote rule set and shared between agents. This enables the design of modular context-aware components within an open and flexible computational system. While DJess enables sharing of knowledge, it does not provide an open model for sharing of data; data is only shared with other DJess agents. The models presented in this thesis have much in common with the vision presented in [91]. These models are developed for context-sharing to enable the construction of a knowledge-base and to encourage reuse. The aim is to create models which are not tied to a specific implementation, but rather provide schemas which can be adapted to any system, which differs from the DJess system.

2.2.4 Rich Communication

Rich communication often includes verbal, visual and textual components, and is commonly associated with video conferencing systems, which also provide group communication. Early prototypes in video conferencing tools include vic [81] and the mStar environment [106]. Today we see an increase of instant messaging [87, 3] and online communication [135] applications being used. These have evolved and now provide chat, audio, and in some cases even video communication. Much research on providing rich communication for mobile devices has been on device and coding-format independence [136], and efﬁcient video coding [75]. This research is based on the notion that rich communication could be handled by a mobile device with limited capabilities. There is however research which aims at using resources in the environ- Thesis Overview 17 ment for achieving rich communication. In [111] a framework called the Composite Device Computing Environment is created for using resources in the environment to overcome the limitations of the small screen in mobile devices. The research presented in this thesis focus on how to provide the rich communication functionality rather than the functionality itself, which sets it aside from research on providing communication tools such as vic and mStar. The underlying implementation could be done using an existing tool which provides enough access to its APIs, such as Marratech [80] which have been used to realise some of the prototypes presented in this thesis. Nevertheless, the ubiquitous communication prototype is closely related to the Composite Device Computing Environment but aims at being able to use any media resources in the environment which can improve the communication experience. Furthermore, dynamic groups presented in this thesis is related to group communication tools, however, it provides the joint beneﬁts of social networking and instant messaging tools as well. Dynamic groups also offer automatic functionality through decision support rules, which is not the case of many of the existing communication tools.

2.3 Research Questions

The motivation for the research presented in this thesis has been to investigate the use of computer science techniques to address the need for remote healthcare provision, where people can receive healthcare services in their home rather than having to move to a hospital or a nursing home. Several problems need to be solved to achieve this, some of which are described in this section. The research questions addressing these problems have been divided into three categories: context-sharing, mediated communication, and new paradigms for healthcare provision. While the research questions all address problems in remote healthcare, they have been kept separate intentionally for clarity.

2.3.1 Problems Related to Context-sharing

1. What are suitable models for sharing of context and rules? It is a known fact that knowledge sharing helps speed up the development of analytical models [89]. Nevertheless, how do we translate this into context- and rule-sharing and how should such models look like to achieve similar beneﬁts as with knowledge sharing? Furthermore, can we utilise context- and rule-sharing for additional beneﬁts other than the development of analytical models?

2. How can we reduce complexity and cost of deploying technology and services into smart environments? Deploying technology and services into a smart environment can be a cumbersome task, not only for the technicians undertaking the work, however, also for the inhabitant who might have to leave their home for some time, and also suffer through the training process with the system. Furthermore, every smart home is unique as it needs to be personalized to the needs and preferences of the inhabitant. How can we simplify this process and make 18 Thesis Overview personalization and deployment of smart environments easier?

2.3.2 Problems Related to Mediated Communication

3. How can we overcome limitations of current handheld communication devices? Although we are seeing more functionality appear in our mobile phones they still suffer from the same physical limitations. They have limited screen size, limited input capabilities, limited video quality, limited battery life etc. How do we overcome these limitations while maintaining mobility and without adding to the current levels of complexity?

4. How can we improve communication within smart environments? Rich communications featuring video, chat, and audio, is becoming more common. Col- laboration often relies on group communication and may require ﬂexible creation of groups where participants may have no prior knowledge of each other. While there are tools for both rich communication and group communication, some functionality can be complex to use, and some desired functionality might be missing. How do we support dynamic group communication in a simple and ﬂexible way, and how can we use smart environments to further improve functionality and usability?

2.3.3 Problem Related to Healthcare

5. How can we improve healthcare using smart environments? Saving costs and improving the quality of the health service may seem like conﬂicting goals. Keeping patients from having to move to the hospital or a nursing home by providing necessary assistance in their home may be the solution. So, can we improve health services using smart environments, and how will this change the way we relate to healthcare?

2.4 Delimitations

Security and privacy are both important aspects when undertaking research in pervasive computing, especially if it is applied to healthcare. Nevertheless, both these areas are signiﬁcant research areas on their own, and have not been a focus of the work presented in this thesis. Same goes for the actual gathering of data by means of sensor technology, information fusion, and learning systems, as well as the actual storage and exchange of data. Furthermore, mediated communication relies heavily on the underlying network infrastructure. Network protocols or architectures are, however, beyond the scope for this thesis and are not discussed in any detail. Thesis Overview 19

2.5 Research methodology

The research presented in this thesis has been conducted with the ambition to solve actual problems found in the real world through the use of computer science tools and techniques. This has been accomplished by working in research projects that involve not only academia but also industrial, research and healthcare organisations. This has resulted in a research methodology that is practical in nature, resulting in prototypes and artefacts which can be used in real life and thus demonstrate the proposed solutions. Prototypes have both given valuable insight about important problems, and have functioned as a source for inspiration and new ideas. The prototypes presented in this thesis can mainly be divided into two categories: smart environment prototypes,andmediated communication prototypes. The smart environment prototypes include different tools for deployment of technology and services into smart- environments, such as a model for the exchange of data as well as a database to go with it, a model for the exchange of rules, and a graphical user-interface for defining rules in smart environments. The mediated communication prototypes include tools for improving communication, such as for utilising different resources in the environment, and for mobile group communication. These prototypes were developed and evaluated to serve as proof of concept systems which prove that theories and ideas work in practice. The problems addressed in this thesis were identified primarily by observation of user needs, user behaviour, and existing services. In attempting to deploy context-aware services it became clear that a lot of work was put into development of totally new solutions. Hence, the need for a common format which everyone could adhere to was raised. Providing such a format would promote the creation of a knowledge repository where information, such as data, rules, and models, could be shared. The methodology used to address this was to explore different data formats for online data representation, and then proceed with developing schemas which would be general enough for reuse within the targeted domain, in this care home-healthcare. Initial evaluation was done by deploying the models and testing them with real data and make sure they addressed the problems they were created to solve, however, wider deployment would be necessary for more thorough evaluation. Close collaboration with nurses at a local nursing-home identified the need for better information dissemination and remote collaboration between personnel. Having been previously introduced to the benefits of video conferencing the nurses agreed that rich group communication (e.g. video, audio, chat) would partly address this problem if the solution was light and flexible enough. This raised the problem of creating powerful communication solutions which would not be cumbersome to carry. The solution was found in the utilisation of smart environments and the resources available therein. Development was performed in collaboration with users for continuous feedback. For the group communication tool, dynamic groups, the models previously developed for common data representation were reused and refined. The prototypes which were developed to address the communication needs were then evaluated by users. 20 Thesis Overview

2.6 Roadmap and Overview of included articles

In this Section, each publication included in the thesis will be brieﬂy presented. The overall theme is summarised in paper 1 with the remaining papers being divided into three different categories: Mediated Communication, the openHome Suite, and Dynamic Groups (see Fig- ure 2.1). The Mediated Communication category consists of work to improve communication using ubiquitous computing and smart environments (paper 2 and 3). The openHome Suite categories include articles developed as a part of a software foundation named the openHome Suite. These articles include XML models, tools and studies related to context-sharing (paper 4, 5, 6, and 7). The Dynamic Groups category include work on a new communication model named dynamic groups which is a novel approach to ﬂexible and mediated group communication (paper 8 and 9). The work on mediated communication raised a need for smart environments which is addressed within the openHome Suite. The dynamic groups use the models in the openHome Suite and may in turn be used to improve mediated communication.

Paper 4: homeML – An Open Standard for the Exchange of Paper 3: Supporting Data Within Smart Environments Automatic Media Resource Selection Using Context- Paper 5: HomeRuleML – A Model for Awareness the Exchange of Decision Support Rules Within Smart Environments

Paper 2: Wearable Systems in Nursing Paper 6: HomeCI – A visual editor for healthcare professional in the design Home Care: Prototyping Paper 1: Remote of home based care Experience Healthcare Monitoring and Assessment Paper 7: Assessing the Impact of Individual Sensor Reliability Within Smart Living Environments

Paper 8: Creating Dynamic Groups using Context- awareness

Paper 9: Profile Management in Dynamic Groups

Figure 2.1: The roadmap of included articles Thesis Overview 21

Paper 1: Remote Healthcare Monitoring and Assessment The ﬁrst paper presents an overview of how healthcare monitoring and health assessment can be performed from a distance. It presents the key motivation as to why research is still required in remote healthcare, and introduces a number of key areas of research. Overall, this article captures the theme of this thesis and shows how telemedicine and telecare can work together with pervasive computing, and how technology and computer science can be used to address some of the problems in achieving remote healthcare. It shows how pervasive computing and communication in smart environments can serve to provide healthcare services in patients’ homes.

Paper 2: Wearable Systems in Nursing Home Care: Prototyping Experience The second paper presents the details of the prototyping of a system for mediated communication amongst nurses in a nursing home. An ethnographical study revealed that improved communication among personnel and remote medical workers is highly desirable. To provide this a prototype is developed using participatory design. Both paper prototyping and "wizard of oz" prototyping are employed in order to involve the end users in the design process.

Paper 3: Supporting Automatic Media Resource Selection Using Context-Awareness The third paper presents the details of a novel approach to achieve rich mediated communication without the need to carry heavy and cumbersome equipment. By transferring media streams between devices in the environment the user can ﬁnd the balance between cost, privacy, and quality that the user prefers whilst providing the delivery of media. The paper also presents an algorithm which can be use to assist the user with selecting which devices in the environment to utilise based on user preference, on cost, quality, and privacy.

Paper 4: homeML - An Open Standard for the Exchange of Data Within Smart Environments The fourth paper presents an open XML model developed for the purposes of context sharing. The model is designed for use within smart environments where sensor data located both in the environment and generated by the inhabitant can be represented, stored and shared. The intention of the model is to share data between developers and research organizations in order to speed up the development of analytical models, tools, and services in addition to gaining a further understanding of the challenges within this environment.

Paper 5: HomeRuleML - A Model for the Exchange of Decision Support Rules Within Smart Environments The ﬁfth paper presents an open XML model for rule representation and sharing. The model is used to deﬁne rules using homeML as the underlying context storage infrastructure. Such rules can then be shared between other environments using the fundamental concepts of the homeML model. The overarching goal of the approach is to decrease time spent on developing, testing, and evaluating rules by providing a repository of rules developed by others which can be re-used and built upon. 22 Thesis Overview

Paper 6: HomeCI - A visual editor for healthcare professionals in the design of home based care The sixth paper presents a visual editor for creating rules which will govern the use of technology within the home environment. It is intended to make an otherwise complex task of generating and capturing rules simple enough for those with the knowledge of what rules are needed and especially taking into account if they do not have a technical background. The editor supports the user by allowing them to drag and drop symbols onto a canvas and to add connections to form logic expressions, which in turn forms the different rules themselves. During evaluation healthcare professionals were asked to use the system to develop and interpret a set of rules according to a predeﬁned set of evaluation criteria. The results show that users with no prior experience of using such a system could use the editor successfully with only a very short period of training (circa 20 minutes).

Paper 7: Assessing the Impact of Individual Sensor Reliability Within Smart Living Environments The seventh paper presents an information management framework to process sensor based data within smart environments. Based on this framework an assessment on sensor reliability on the classiﬁcation of activities of daily living is presented. From this assessment it is possible to identify which sensors within a given set of experiments can be considered to be the most critical and as such it becomes possible to consider how this information may be used for managing sensor reliability from a practical point of view.

Paper 8: Creating Dynamic Groups using Context-awareness The eighth paper presents a new model for communication named dynamic groups. The definition reads "The dynamic group model defines group communication where the participants can be invited based on context and social networks in both physical and virtual settings". Dynamic groups can therefore be considered as a flexible communication model intended to simplify mediated group communication. In this paper the overview of the architecture and necessary components for dynamic groups are presented, which includes the models presented in paper 4 and paper 5.

Paper 9: Profile Management in Dynamic Groups The ninth paper presents a group management model for dynamic groups as well as a proposal for privacy handling. The group management model, named HomeCom, is an open XML model which integrates with both homeML and HomeRuleML presented in paper 4 and 5. This is an essential component in dynamic groups with the purpose of managing the dynamic groups as well as profiles and queries on groups and/or profiles. The presented privacy solution relies on a layered structure where the user can define granularity and choose which information is presented to different groups. Thesis Overview 23

2.7 Smart Environments for Remote Healthcare

In this section, the topic of smart environments for remote healthcare monitoring and provision will be presented. Based on the papers summarized in the previous section, two different aspects will be considered, namely context-sharing and mediated communication. Remote healthcare has been selected as the motivation for the research given that context-sharing and mediated communication are both important elements in achieving and deploying remote healthcare services. Nevertheless, the research has uses in many areas and should not be considered limited to the healthcare environment.

2.7.1 Context-sharing and the openHome Suite

The idea of creating a model for context-sharing was inspired by a common platform for the sharing of ECG data for comparison and data analyses [145]. Due to how well the concepts of homeECGML was received, work started on creating the openHome Suite, a collection of open XML-models used to specify how sensor-data should be stored and handled, as well as some tools which can be used to utilise this data. The goal of the work was to create a set of utilities and tools which could easily be deployed within the context of smart environments in order to quickly achieve desired functionality, and which would result in data that could benefit other test-sites as well. The openHome Suite began with the concept of homeML [96] which was designed as a potential model to promote a common way of storing data within smart environments with the foresight of establishing a central web based repository for use by the general research community. A common data repository can have many benefits, data can be used for compar- ative purposes, however, can also be used to help pre-train neural network type systems which in turn help decrease deployment time. homeML provides detailed information about sensors (sensor events, sensor location, etc.), the environment (room information, room purpose, etc.) and the inhabitant (including any wearable sensors). homeML is built using XML as it is an accepted standard for sharing of data over the Internet. Figure 2.2 shows an exerpt of the nodes in the homeML definition. Apart from general information about the smart-environment and the rooms, it has a Device subtree which stores information about various sensors in the environment. Each device is identified by a deviceID, and contains a general description, information about location, device type (e.g. MovementSensor, TemperatureSensor, etc.), and which unit is being used for the sensor-data (e.g. Celcius, State, etc.). Furthermore, there is information which may be used for separating between different trials or "runs" (RealTimeInformation) containing sample rate of the sensor, as well as start and stop time. Finally there is room to store sensor data in the Event node. Each event is identified by an eventID, however, also contains a time stamp. What is not shown in Figure 2.2 is the Inhabitant subtree. This looks similar to the Room subtree, however, also contains information about the inhabitant and devices carried by the inhabitant. Following definition of the XML model work began on creating tools that would simplify the deployment of smart environments using homeML as a context-storage infrastructure. First a database implementation was created which featured a simple user interface for creating new smart home entries along with the ability to add, edit, and remove data. Furthermore, 24 Thesis Overview

SmartHome homeID + Must exist at least once + * Can exist any number of times Room roomID ? Optional * Nodes without any sign must exist once Device deviceID ? Description ? DeviceLocation

XPos YPos ZPos

DeviceType

Units

* RealTimeInformation runID

SampleRate StartTime EndTime

+ Event eventID

TimeStamp Data

Figure 2.2: Excerpt of the homeML definition [96] a simple browser with rudimentary analytical capabilities was created (see Figure 2.3). This browser has the ability to provide an overview of the data and by marking desired sensor events the user can choose to plot the events to get a better view and to compare sensor data. With a context-storage infrastructure in place (in the form of homeML) the work contin- ued with the definition of HomeRuleML [51]. HomeRuleML is a model for defining rules to be used within smart environments, and like with homeML the goal is to provide Home- RuleML data (i.e. rules) within the general research community for the purposes of representation, storage and exchange. HomeRuleML uses data stored within the homeML format, and therefore benefits from the advantages of a common data repository. HomeRuleML is an open XML model which uses sensor- and context-data which has been stored in accordance with the homeML model. This means that HomeRuleML rules can focus on specifying the rules by defining which sensors and contexts are of interest, instead of dealing with the data retrieval etc. The model is recursive which enables advanced logic expressions and the reuse of other, more generic, rules such as a general rule for detecting if food has been taken. Common logical operators (AND, OR, NAND, NOR, XOR, NOT, and a special operator ORDER) were used to form the expressions, and common condition operators (greater than, less than, equals, not equals) were used to set conditional expressions. A simplified example of a rule for detecting if food has been taken (it is assumed that movement in the kitchen, and the opening of the fridge door is enough) can be seen in Figure 2.4. An important component for the acceptance of the openHome suite is a graphical frontend system. For defining rules within a smart environment there is the HomeCI tool [93]. HomeCI is a simple tool designed for non-technical people who know which rules should be in place, however, are not able to actually write the software or express the logic which would be required from a computational perspective. It can help carers and non technical personnel define rules and actions the smart environment should take. It consists of a user interface (see Figure 2.5) where a user can drag symbols onto a canvas and connect them to form logical Thesis Overview 25

Figure 2.3: The homeML browser showing movement in the office landscape from one day in the office expressions and rules. Although HomeCI has not been fully integrated with other tools in the openHome Suite, it is believed that this type of graphical user interface will significantly improve the way smart environments are personalised to the individual. To test the HomeCI tool a simple exercise was performed by four users, two from the Northern Sweden, and two from Northern Ireland. All users were experienced in working in the domain of elderly care and had good appreciation of the needs of technological solutions to be deployed in the home environment to support independent living. Following a short period of training with the system, the users were asked to construct rules, as well as interpreting pre-constructed rules. The initial results from this evaluation show that with a relatively short amount of training, healthcare professionals were able to utilise the concepts of the HomeCI editor. The HomeCI prototype demonstrates how it is possible to use visual notation to allow those with a nontechnical background to manage the deployment of technology within the home environment. 26 Thesis Overview

Food Sensor 2007-04-23 11:58:05 Kitchen Movement Sensor Movement Sensor State Movement Fridgedoor Sensor DoorSensor State Open Food Taken

Figure 2.4: The XML-representation, using homeRuleML, of a Food Sensor rule

As can be seen in Figure 2.5 the symbols depict mainly objects and activities which are either taking place or are expected to happen. Determining ADLs is a very important aspect of smart environments. This is typically undertaken by deploying a number of sensors expected to be relevant in determining an activity, then analyzing the data in order to ﬁnd sensor data patterns for the different activities. Once a pattern has emerged a rule can be created, for instance using HomeRuleML, and deployed in the smart environment. Nevertheless, it may be proposed that some sensor data may not be as relevant to the activity as other sensor data. An experiment was set up in order to test the impact of different sensor data in determining ADLs and how the system would perform in instances of sensor failure. The experiment had as goal to assess if a drink was being prepared, and if so also determine which drink was prepared. Seven different sensors were placed in the environment (see Figure 2.6): a fridge sensor, a cupboard sensor, a coffee sensor, a tea sensor, a sugar sensor, a water sensor, and a kettle sensor. The analysis of the data recorded from this study shows that certain sensors have a greater impact in determining the current activity than others and in addition, failure of certain sensors will have a greater impact in terms of the overall classiﬁcation of activities in comparison with some others. Results from the study also demonstrated that the activity can be inferred even if some sensors fail. In a healthcare environment this can be deemed as a positive result, as these results indicate that the system can still be used even in instances of limited system failure. Thesis Overview 27

Figure 2.5: Screenshot of the HomeCI visual editor

2.7.2 Mediated Communication and Dynamic Groups

Remote healthcare monitoring and provision relies on health monitoring being possible to do from a distance, by offering rich communication between patient and healthcare personnel, and being able to cater for patient needs in their own home. Providing a means of rich communication can be problematic since both patient and healthcare personnel need access to good communication equipment. Nevertheless, healthcare personnel are likely to be moving about in a hospital or nursing home which raises the need for solutions which are either mobile, or available in the surrounding environment. The ﬁrst approach to solve this problem was to create a mobile solution for rich communication. This was created in collaboration with nursing staff in a nursing home, through the use of rapid prototyping. The nurses who participated in the design phase would normally spend a lot of time gathering information from personnel who had worked during the night or the weekend to ﬁnd out if there had been any problems. Much of this time would also be spent trying to explain certain health conditions to experts, which sometimes ended up in the provision of confusing messages. The prototypes developed in collaboration with the nurses aimed at making the desired information available to the nurses whenever they needed it, and they 28 Thesis Overview

(a) Cupboard with door sensor (b) Kettle with tilt switch and contact switch on tap

Figure 2.6: Sensors within the smart laboratory to assess the impact of sensor failure on ADL classification aimed at simplifying communication and reducing confusion. In this work a "wizard of oz" prototype was produced. A "wizard of oz" prototype functions through the input of a control person who mimics certain behaviour to produce the results of a finished product. The "wizard of oz" prototype presented in this work consisted of a simple wearable computer (see Figure 2.7) and used e-meeting software which provided all the necessary tools for demonstrating the concept. The "wizard of oz" would provide all the desired information via the virtual whiteboard and would intercept all phone-calls when the nurse was busy. Instead of trying to describe a condition to an expert the nurse could now send a live video stream which in many cases reduced confusion. Though the physical representation of the prototype was not perfect it did address the actual needs identified during the interviews with the nurses and during the design phase with the paper prototype. The second approach to solving the problem of achieving rich communication whilst moving about a lot required the provision of communication capabilities in the environment. Assuming the user is located in a smart environment with a multitude of media devices the system would have to suggest the most suitable media resources to use, at the most appropriate time, to make the system more usable and help the user select which devices to use for mediated communication. An algorithm was created to address this need. It decouples the selection problem into smaller parts which can be solved independently. In order to com- Thesis Overview 29

(a) Wearable computer outﬁt: battery pack and (b) A nurse wearing a computer and head- VGA converter for the head-mounted display mounted display while attending a patient. (left), Sony U70P computer (lower center), Blue- tooth headset (upper center), and M2 head- mounted display (right).

Figure 2.7: The "wizard of oz" prototype.

Media source Selected media Media sink resources Infrastructure Media sink Media streams Media to/from other gateway users Media sink Media source

Figure 2.8: A conceptual communication model. pare resources to each other, abstractions for privacy, quality, and cost are introduced. This approach raises many research questions, such as which contexts should be used to form the privacy, quality, and cost abstractions, how is it possible to compare media resources to each other, which media resource should be selected and when to change to a different one, and how can privacy be ensured. Figure 2.8 illustrates the created conceptual model showing two communication abstractions, media source and media sink. A media source is an abstraction over a media-capturing hardware device, such as a camera or microphone, and the software that generates the media stream. After being generated, the media stream is distributed to the recipients via a network infrastructure. When the media stream has been received and processed by the recipients, a media sink presents the media stream to the user. A media sink is an abstraction of a media- rendering hardware device such a loudspeaker or a display and its accompanying software, which is used by the device to send or receive the media stream. 30 Thesis Overview

A system which can utilise any media sink and media source in the environment can adapt to the location of the user by selecting the closest device. This is very useful in emergency situations where a patient may be unable to move to a designated location to establish contact with a remote carer. By providing the most suitable media resource a healthcare professional may still be able to assess the severity of the situation, however, to achieve this functionality the system would have to establish communication sessions automatically, something which is possible through dynamic groups. Dynamic groups is a new concept of looking at group communication. Dynamic groups focus on availability and ease of use, and being able to find information and competence at the user’s terms. Classical group communication tools would normally be limited to members of a contact list, or possibly to a room such as a forum or chat-room. While dynamic groups have many similarities to classical group communication tools they do offer new functionality such as the possibility for automatic group formation based on predefined rules. Simply put, the dynamic group model combines the strengths from previous group communication models to achieve something which is more powerful, more flexible and easier to use. The dynamic group model is defined as "group communication where the participants can be invited based on context and social networks in both physical and virtual settings". This model implies that virtual group communication can happen anywhere, either in a physical setting where people meet face-to-face, or in a virtual setting. Although dynamic groups are not only for use in healthcare, the communication model is well suited for this domain as it can be used in conjunction with smart environments and can avail of predefined rules and inputs from a sensorised environment to automatically set up group communication. To prove the concept a prototype was developed for a Nokia 6131 NFC edition phone (see Figure 2.9). The concepts of the Dynamic groups have been built on the openHome Suite (see Figure 2.10), and have utilised both homeML (for the purposes of context storage), and Home- RuleML (to avail of the rule engine). It also utilises a profile management system referred to as HomeCom, which handles information about users and groups, and also provides a basis for advanced search queries. The profiles contain necessary information about users and groups for search queries, but also have a built in privacy filter where users can specify which information should be visible and accessible to others. In summary this approach presents a model for the exchange of profile data in addition to the presentation of a model for the handling of privacy within dynamic groups. The solutions on mediated communication presented in this thesis make it possible for healthcare personnel and the user to stay in touch through a form of rich communication and also help the user to maintain a social network outside the home. This is a key feature to ensure there is a notion of continuum of service delivery once the person leaves their home and that they do not feel as if they cannot leave their own home. If desired a rich communication system can be integrated with a smart environment and programmed with rules, thus making the system more autonomous and user friendly. Combining dynamic groups with automatic media resource selection and mobile technology creates a powerful combination for mediated communication in smart environments. Thesis Overview 31

Figure 2.9: A screenshot of the dynamic group prototype running on a Nokia 6131 NFC edition mobile phone

Dynamic Group users

HomeCom

XML data (queries & result) L eM m Ho Sensor data (contexts) Sensor data

Rule data (new rules & result)

HomeRuleML

Figure 2.10: An overview of the components which form the Dynamic Group concept 32 Thesis Overview

2.7.3 Conclusions

The roadmap in Figure 2.1 presents how the included articles are tied together in this thesis. It also shows three areas, namely "Mediated Communication", the "openHome Suite", and "Dynamic Groups". These areas of research are important as highlighted in Part 3, as new knowledge can help establish and develop remote healthcare which makes it possible for patients to receive care in their home. Mediated Communication is an area which helps enable communication over distance. This is addressed in this thesis by two different solutions, namely a wearable computer prototype (presented in Part 4) and a prototype for utilising resources in the environment for communication (presented in Part 5). The wearable computer solution addresses the communication needs for healthcare personnel by providing a wearable computer solution which enables communication through a bluetooth headset, head-mounted display, and a camera. The prototype for utilising resources in the environment include an algorithm which selects the most suitable resource to use by comparing the resource’s privacy, cost, and quality values to the user preferences. This solution, however, require the use of a smart environment where resources specify privacy, cost, and quality values and are configured for dynamic use. The openHome Suite contains a number of proposed models addressing problems in smart environments, such as the complexity of deploying technology and services into smart environments. The deployment complexity is addressed in three different ways, by sharing context data (presented in Part 6), by sharing rules (presented in Part 7), and by simplifying the creation of rules (presented in Part 8). Sharing contexts and rules reduce the time it takes to develop models and services and can improve the quality of them, since more available data means there is more data to base models and services on. In Part 8 a graphical user interface for creating rules is presented. It is intended to simplify the complex task of creating rules so the people who know what rules are needed can help create them. The openHome Suite also addresses another problem, and that is sensor reliability. This is addressed in Part 9 where the impact of individual sensors to infer a daily activity is explored. Dynamic Groups addresses problems in group communication and collaboration, such as the complexity and flexibility of establishing group communication. Group communication is important for collaboration in general and for providing remote healthcare services in particular. A solution for the complexity and flexibility problems is presented in Part 10 which presents the new communication concept called dynamic groups. The solution for solving flexibility problems is further described in Part 11 which presents a profile management system as well as a privacy solution for use in dynamic groups. The research presented in this thesis has been proposed, implemented, and evaluated to prove feasibility. It is motivated by the need for a paradigm shift in healthcare where care is provided in the patient’s home. This introduces the need for remote healthcare in which mediated communication and smart environments are necessary. Part 3

Remote Healthcare Monitoring and Assessment

Citation: Chris Nugent, Dewar Finlay, Richard Davies, Mark Donnelly, Josef Hallberg, Norman Black, David Craig, "Remote Healthcare Monitor- ing and Assessment", To appear in Journal of Technology and Health Care, 2008.

Contributions: This article was achieved together with Chris Nugent, Dewar Finlay, Richard Davies, Mark Donnelly, Norman Black, and David Craig. The main contributor to this article is Chris Nugent. My contributions are primarily in the telemedicine section as well as the section for ambient and pervasive computing, where my experience in mediated human-to- human communication and pervasive computing was availed off. I also contributed to the general structure and content of the article along with its presentation.

Remote Healthcare Monitoring and Assessment 35

Remote Healthcare Monitoring and Assessment

Chris Nugent1, Dewar Finlay1, Mark Donnelly1, Josef Hallberg2, Norman Black1, David Craig3 1 School of Computing and Mathematics, Faculty of Engineering, University of Ulster, Northern Ireland 2 Department of Computer Science and Electrical Engineering, Luleå University of Technology, Sweden 3 Belfast City Hospital / Queens University of Belfast, Northern Ireland

3.1 Introduction to remote healthcare monitoring

Remote healthcare monitoring is the process of assessing the well-being of a patient when the patient themselves and their healthcare professional are not physically together in the same room. Conventionally, patient assessment in Primary and Secondary healthcare provision involves a face-to-face consultation between the patient and the healthcare professional (General Practitioner, Consultant etc.). Advances in technology, specifically medical devices, sensors and high speed fixed and wireless communication networks have now made it possible to bring the assessment process to the patient, as opposed to limiting the assessment to the constraints of hospitals and doctors’ surgeries. As a result, it is now possible for a patient to be assessed whilst in the comfort of their home or to have their vital signs monitored whilst, for example, shopping or at work. In addition, it is possible for patients to visit their local Doctor’s surgery and benefit from expert consultants and receive their advice, without having to have a face-to-face meeting with them. It is the aim of this review to introduce the concepts of remote healthcare monitoring, discuss the drivers which have lead to the realisation and uptake of this approach and to look at the benefits that evolving technologies can offer in this domain.

3.1.1 Changes in Population Demographics

We are at present witnessing a change in the way our population is represented, in terms of age profile. In the 1950s the largest percentage of the population was made up by those in the category of 0-9 years of age. At the turn of the century this category was amongst one of the largest, with a similar level of numbers being found in other age categories. Predictions for the next thirty years have shown that this trend will continue and infact result in a profile of age categories with almost similar sizes across all age groups (Figure 3.1). People are living longer and the total population size is increasing. In addition, the percentage of our population represented by those aged 65 and over is steadily increasing [2]. From a healthcare perspective this offers a number of challenges. In the first instance, we are faced by the simple fact that the population is now larger and hence places a larger demand on healthcare services. 36 Remote Healthcare Monitoring and Assessment

Secondly, there is an increased prevalence of healthcare problems and instances of disease within the elderly and thirdly there are now fewer younger people to care for the elderly [115]. Taking all of this into account has resulted in healthcare providers, governments and members of society, searching for new paradigms of healthcare delivery.

Population Catergories 85+ 2030

70-80 2000

40-50

1950's

0-10 Population (Males) Population (Females)

Figure 3.1: Estimated proﬁle of age categories in 1950, 2000 and predictions for 2030.

3.1.2 The impact of Technology from social and economic perspectives

We have previously introduced the concept that the increase in the size of the population has a direct impact on the prevalence of instances of healthcare problems and of long term chronic diseases. In addition to placing a resource burden on primary and secondary healthcare organisations, this increase in population size has additional societal and economic implications. There are benefits from both perspectives which would support the introduction of remote healthcare monitoring. From a societal perspective, people are wishing to play a more active role in their own healthcare. This can be facilitated, at least notionally, via the realisation of remote healthcare monitoring where the patient is given a higher degree of responsibility to become more proactive with the management of their own healthcare and its assessment. Involvement in such a manner can also lead to the patient having a greater understanding of not only their condition, but also its means of assessment. In addition, there is the general appreciation that people recover more quickly in their own homes and would prefer to remain in their own homes, even if they are at greater risk [28]. Remaining at home for an extended period of time can also be linked to improvements in the patient’s perceived quality of life. Taking this into account, the use of remote healthcare monitoring has the potential to avoid instances of institutionalisation when the patient requires high levels of healthcare assessment. It can also avoid lengthy and inconvenient trips being made for the purposes of routine clinical assessments. Patients can also benefit from remote expert clinical diagnosis. For example, it is Remote Healthcare Monitoring and Assessment 37 possible to send a patient’s clinical details from one institution to another and receive expert clinical assessment via remote means. In addition to the societal benefits, deployment of technology to support healthcare assessment via remote means has a number of economic benefits. If a person can have their healthcare assessed within their own homes, removing the requirement of temporary or permanent institutionalisation, then significant healthcare savings can be gained. Additionally, those patients receiving clinical care within a hospital setting can be discharged in a much shorter space of time and continue to have routine assessments conducted via remote means. Finally, consultants can offer remote consultations to patients all over the world without having to leave their own office. This offers benefits to the patient in terms of them receiving treatment from ’world class’ specialists who, because of geographical distances, would otherwise not be able to provide such treatment.

3.1.3 Models of Patient-Healthcare Professional Interaction

Traditionally, the means by which those involved in healthcare provision interact with their patients is based on a bi-lateral communication model (Figure 3.2(a)). In this respect it is normal for the patient to have direct contact with a number of healthcare providers, however, commonly there is a potential lack of a uniﬁed infrastructure to support communication between healthcare providers. Deployment of technology and the establishment of care models facilitates communication channels to be established between both patients and healthcare providers as well as intercommunication between the healthcare providers (Figure 3.2(b)). This offers an improved means of healthcare delivery from the patient’s perspective as all those involved in the delivery of their care can interact.

Physician

Patient

Carer Physician Healthcare Professional

Healthcare Patient Pharmacist Professional

(a) Interaction between stakeholders: the classic (b) Enhanced stakeholder interaction facilitated situation via technological communications

Figure 3.2: Examples of Patient-Healthcare Professional interactions.

Although the aforementioned care models can be used to facilitate the deployment of technology, their conceptualisation can at times be faced with resistance. The deployment of technology within the realms of healthcare provision can offer benefits to the patients, 38 Remote Healthcare Monitoring and Assessment however, the changes of practice which may be required from the healthcare providers’ perspective, in addition to the problem of new service integration, creates major challenges concerning the widespread deployment of the technology. On the one hand, drivers such as the benefits of improved treatment results and alleviation of the problems associated with the shortage of physicians and nurses encourage such developments. Nevertheless, such innova- tions are faced with the constraints such as technology costs, reimbursement issues and legal issues, to name but a few. This inevitably results in a ’push-pull’ scenario (Figure 3.3) from a technological perspective. The development of new technology results in new paradigms being pushed into the healthcare market. These can be considered as high risk solutions. If, on the other hand, a user driven approach is adopted, a pull effect can be witnessed. This may be viewed by some as the desired effect, where the end solution will be based on mature technology, which, from a technological perspective may not offer high levels of innovation, however, from the healthcare perspective can be seen as a low risk solution meeting the needs of the end users. In order to provide the patient with the best solution to their needs, a user centered design process offers many benefits. Adopting this approach allows the patient to be involved during the early stages of the design process.

s n o ti a c li p p Push Effect A High risk enabling w e N new solutions

Pull Effect y g Driven by user requirements and lo o n mature solutions h c e T w e N

Figure 3.3: "Push-Pull" effect of technology within healthcare market place.

3.1.4 The impact of the Internet on Healthcare Delivery

The Internet is one of the most widely used and accepted forms of modern day communication, as well as a rich source of information. Patients can now seek healthcare advice, research Remote Healthcare Monitoring and Assessment 39 their symptoms and determine possible diagnosis, in addition to researching available courses of treatments. Additionally, the Internet can be used as a forum to support communities offering discussion groups dealing with health and healthcare topics. Furthermore, the Internet offers great potential for distributed healthcare, which is important especially for sparsely populated rural areas, and for people with limited mobility. The Internet makes it possible for user-centric healthcare, where the user’s needs are in focus. More demands are also placed on being able to locate the necessary information and help in user understanding. This has created more demands on information and communication resources, as well as on available healthcare professionals. Nevertheless, this also means that for many health-related issues, the user can receive help from centralized healthcare resources. For example, a parent concerned about a child’s skin rash could be taught how to acquire high-resolution pictures of the affected area and subsequently forward such pictures to a centralized healthcare service. This approach would offer a fast and efﬁcient means of receiving direct feedback regarding the condition of the skin, without the necessity of the parent and child having to travel to the hospital for a physical examination.

3.2 Telemedicine

This Section aims to introduce the notion of Telemedicine and how technology can be deployed to link healthcare professionals together in an effort to improve the quality of the healthcare delivery process.

3.2.1 What is Telemedicine?

Telemedicine has received many deﬁnitions over the years. Most notably it can be described as the remote use of medical expertise at the point of need [11]. This results in the use of telecommunications to facilitate the sharing of medical knowledge and the delivery of healthcare over a distance [43]. With this approach medical expertise can be delivered to remote sites, rural areas or understaffed regions. Although Telecare may also be considered as a form of Telemedicine, for the purposes of this article we will treat these two topics individually and focus within this Section on the element of Telemedicine and the associated technology which fosters a collaborative working environment between medical experts.

Telephony

A rudimentary means of providing telemedicine is via telephone communications. This can be in the form of two specialists discussing a patient’s diagnosis or prognosis over large distances or similarly can relate to a consultation between a patient and a specialist. Other examples include automated services which patients access, via the telephone, to provide details regarding their current state of health. An example of this type of system is an automated diabetes service which permits patients to verbally enter information regarding their condition via an automated telephone system [12]. Such a system facilitates the routine assessment 40 Remote Healthcare Monitoring and Assessment of a patient’s condition without the need for direct consultation thus alleviating pressure on healthcare services.

Video Conferencing Systems

Video Conferencing Systems have become one of the most popular platforms used to facilitate telemedicine and subsequently improve communications between healthcare professionals [37]. Such systems typically include the ability to send and receive video, audio, and text between one or more users. Additionally, some systems provide tools, such as shared whiteboards, which facilitate the communication of ideas and thoughts via informal means. The main purpose of using video conferencing systems in telemedicine is to provide a sense of presence through the provision of visual instructions, or demonstrations to patients, and to visually present or describe problems and symptoms to healthcare professionals. Video conferencing provides a wide range of services that healthcare providers can avail of for example nurses can use video conferencing to make house-calls while keeping contact with the doctor, whilst patients can use the system to set up meetings with healthcare professionals from their home, instead of having to travel to the hospital.

Point-to-Point systems

Similar to video conferencing, point-to-point systems offer commutations between stakeholders, however, they only provide communications between two points. At present, several such systems are freely available. While these systems are often sufﬁcient in providing simple communication between a patient and the doctor, they do not offer the support group functionality that video conferencing systems do.

Remote Surgery

At the other end of the spectrum from telephony systems exists the concept of remote surgery. This technology provides the means to support the delivery of remote care through robotics [121]. An obvious advantage of this approach is that it permits world renowned surgeons from across the globe to provide patient care. The ﬁrst remote surgery was conducted in 2001 and was referred to as "Operation Lindbergh". Through the use of three robotic arms setup in an operating theatre in France, surgeons located in New York where able to remotely perform a gallbladder operation over a ﬁbre optic connection. Since this date, several other remote surgical procedures have taken place.

Store and Forward Services

In addition to real-time telemedicine systems such as those already described, offline telemedicine is also possible. Such care delivery does not require both parties to be present at the same time of communication. For example, technology within one consultant’s office can be used to record some patient information which can be subsequently stored and forwarded at a set time to be reviewed offline by another consultant. Remote Healthcare Monitoring and Assessment 41

3.3 Telecare

This Section aims to introduce the notion of Telecare and how technology can be deployed to link remote patient information with healthcare providers in an effort to improve the quality and the healthcare delivery process.

3.3.1 What is Telecare?

Telecare combines the usage of sensing devices and telecommunications to support the remote monitoring of a patient within their home environment with the goal of providing a means of care support. Therefore, via remote patient monitoring it is possible for a Telecare service to react to abnormal situations which are a cause for concern. In response, a Telecare service can issue an alert to a caregiver or a family member. It is also possible for the person themselves to raise an alarm in instances when they may require support, for example, following a fall.

Technology platforms available for home based monitoring

To provide a home based monitoring system to patients who require additional health care services requires exploiting current technology platforms that are already in place. These should be widely available and practical in terms of cost, performance and speed. The Public Switched Telephone Network (PSTN) is a commonly used means of supporting the transmission of information from the patient’s home to a healthcare organisation. The PSTN is available throughout the world and although originally based on analogue lines, is now almost entirely digital at its core. The primary use of the PSTN is the ability to connect people together to provide a phone based voice service. However, in more recent times, the PSTN has been used as a digital communication channel allowing information to be transferred to and from patients’ homes. The PSTN itself has a number of drawbacks in comparison with more recent technologies based on the same infrastructure. It can be more difﬁcult to set up and is expensive to run if the application is resource intensive and operates at relatively slow speeds making it suitable for only a small percentage of applications. Broadband technology is another possibility. As its name suggests, broadband technology offers a wider band of information. One example of broadband is a Digital Subscriber Line (DSL) which encodes digital information at a higher frequency on one channel and sends voice on a lower frequency forming another voice based channel. DSL technology offers high availability, as it is based upon the already existing PSTN system which has worldwide coverage. There are a number of advantages with broadband technology. Firstly, it is easy to set up and easy to use and can even support wireless connectivity. Although dependent on line conditions such as the distance from the exchange, the speed and reliability are greatly improved. Broadband technology, and DSL in particular, are essential to providing the underlying communication infrastructure to allow home based services to become successful. 42 Remote Healthcare Monitoring and Assessment

Telecare Services

One of the most common forms of Telecare services exist in the form of alarm based pendants. Alarm based pendants are essentially a device worn by a patient which can be used in either a passive or active manner. Their primary use is within the home environment and can be used to trigger various alarms. A passive pendant is one which involves no interaction between the patient and the device; this could take the form of a fall detector which passively monitors the orientation of the patient during home based activities. Falls are common place among elderly patients living alone and are the leading cause for such people having to enter into permanent institutional care. Such a device could alert a relative or health care personnel to a serious situation. A more obvious and active approach of an alarm pendant is one that is activated by the patient whenever a security situation arises. Once the pendant has been activated, an emergency message is relayed to a healthcare professional or family member so that the necessary intervention can take place. In addition to pendant alarms, Telecare services can be based upon a number of devices which may exist within the home environment, for example medication management devices, cognitive prosthetics, ﬁre/smoke/water alarms and various door and device interaction sensors. The exchange of physiological data recorded from the patient in their home and transferred to a remote site can be referred to as Telehealth. Although the patient and the communication infrastructure for both Telecare and Telehealth are the same, it has been usual in the past to keep these two terms separate. Typical parameters which can be monitored within the auspices of Telehealth include blood pressure, electrocardiogram, weight and blood glucose levels.

3.4 Ambient and Pervasive Computing

The way in which healthcare can be delivered is changing rapidly. The convergence of mobile communications, decision support systems and Internet computing are offering a wealth of new healthcare paradigms. In this Section, we focus on a number of new and emerging technologies and demonstrate how they can be deployed within the realms of home based healthcare.

3.4.1 Intelligent Homes

Perhaps the most notable advance in care delivery has been through the introduction of the Intelligent Home environment [4]. This environment aims to promote the use of embedded technology to support a person within their own living environment. The desired goal is to offer a means of independent living and extend the period of time a person can remain in their own home prior to institutionalisation [115]. Within such an environment, it is common to find two types of devices; sensors and actuators. Sensors are the devices which can record information about the person in the environment. These may be motion sensors detecting which room the person is in, or they may be pressure sensors detecting whether the person is in, for example, a chair or their bed. Other types of sensors include temperature sensors, Remote Healthcare Monitoring and Assessment 43 door sensors, water sensors, appliance sensors etc. Such sensors can provide sufficient information so that the current status of the person can be inferred and subsequently be reused to recommend how the environment should be modified or controlled. The environment itself is managed through the use of actuators. These can, for example, control the ambient temperature if the environment becomes too warm, or raise an alarm if a hazardous situation arises. For example, a person turns on the cooker and then turns on the taps in the bath. The challenge at present is to find the correct balance between the information gathered by the sensors and the ensuing processing to support the dynamic change of the environment through the use of the actuators to support the changing needs of the person.

3.4.2 Wearable Systems and Smart Clothes

The ability to monitor a number of vital signs from a patient in a pervasive manner has been realised through the introduction of Wearable Computing or Smart Clothes [5]. Smart clothing is a result of the consolidation of sensor technologies, advanced textiles and information and communication technologies [97]. Sensing devices can be embedded into clothing to offer a means of recording information such as heart rate, perspiration levels, body motion / movement and breathing rates. Smart clothing offers the ability to record from a larger sur- face area on the body, for example the torso, hence the limitations of taking recordings from the ﬁnger or the wrist can be avoided [15]. It is possible, during the manufacture of the garment, to fabricate the sensors by either coating a textile with a form of sensing material or to form a sensing material which can then be knitted or woven into the garment. Although many challenges now exist regarding which are the optimal places to locate the recording electrodes [36] and what information should be recorded, smart clothing provides a realistic solution to continuously monitor the person in an unconstrained manner and either provide a means of local processing or relay the information to a central location for assessment by healthcare staff. Smart clothing has huge potential in conjunction with the aforementioned intelligent environments as an augmented means, not only to assess the activities of the person, but also, their health status.

3.4.3 Data Processing and Context Awareness

We have described a number of paradigms whereby information relating to the patient can be shared between medical professionals or can be viewed remotely by a medical professional to offer a means of support. Given the ability to store large amounts of data about the person, their activities and their general health conditions through a number of vital sign markers, it has now become possible to consider deploying data processing systems with the ability to automatically understand the current status of the person. One of the biggest areas of interest at present is the ability to monitor changes in lifestyle and adapt the technology within the environment to support the changing requirements of the patient. As such, much effort is being directed towards the development of data processing systems which aim not only to detect trends within a person’s behaviour but to try to understand the cause of such behaviour and attempt to correlate this with other social or environmental behaviours. 44 Remote Healthcare Monitoring and Assessment

In addition to data processing, the introduction of Context Aware computing within healthcare and intelligent environments has also become more prevalent [66]. The term ’context’ refers to any information which can be used to characterise the situation of a person or computational entity. Context aware computing is used to extract, interpret and then use contextual information in such a way that the system or environment can adapt its current state of operation to match the current context of its use.

3.5 Case study: Home Based Medication Management

We will now show how some of the aforementioned topics have been addressed in a real project scenario - Home Based Medication Management. Within this context we present an exemplar of how the problem domain has been identiﬁed, how a technical solution has been deployed and how emerging technologies have been integrated into the second generation of the system.

3.5.1 Problem Domain

It is a well recognised problem that patients do not fully adhere to their medication regimen having received a prescribed course of medication. This results in a signiﬁcant burden being placed on the healthcare service due to extended patient welfare and healthcare costs. Accord- ing to the World Health Organisation (WHO) adherence is deﬁned as "the extent to which a person’s behaviour - taking medication, following a diet, and/or executing lifestyle changes, corresponds with agreed recommendations from a healthcare provider" [150]. There have been numerous attempts to quantify the impact of non-adherence to medication, including:

• one third of people take all their medication, one third take some and one third do not take any at all. • as many as 125,000 deaths may occur each year in the US as a direct result of non- adherence to medication. • up to 23% of admissions to nursing homes can be attributed to non-adherence. dherence to medication is a complex issue and stems from problems such as the patient not understanding their medication regimen, perceived side effects and ﬁnancial issues [13]. These can also be further complicated by any confusion which may arise through the context in which the medication is prescribed, delivered and taken by the patient.

3.5.2 Potential deployment of technology

Given the huge impact associated with non-adherence, many efforts have been made to deploy technology as a potential means to alleviate the effects of the problem [95]. Technology has been deployed in three different manners. In the ﬁrst instance, pill holders can be offered to the patient which store the medication in labeled compartments for various times throughout Remote Healthcare Monitoring and Assessment 45 the day. The second type of device is an extension of the ﬁrst, however, electronic modules support the inclusion of pre-programmable alarms. The third is a monitoring device which provides a remote means to assess if the patient has taken their medication. This last type of device will be the focus of this Case Study.

3.5.3 The MEDICATE system

The aim of the MEDICATE system was to develop an Internet based care model and associated peripherals to support the needs of all stakeholders within the prescribe to intake chain of medication. The anticipated beneﬁts of the system were the ability to improve and support the patient with the management of their medication in addition to providing a means to support communications between all stakeholders. To achieve this vision it was necessary to develop a suite of interfaces which could be used by each of the stakeholders. The interfaces took the form of both custom electronic devices and software interfaces all connected via an Internet based care model as shown in Figure 3.4.

Figure 3.4: Overview of MEDICATE care model to support home based medication management.

This care model supported communication between the patient within their home and their remote formal or informal carer, in addition to a communication channel between healthcare professionals. Figure 5 below shows the various stakeholder interfaces. Brieﬂythese include:

• Patient: a mobile medication management system (Figure 3.5(a)) and a base station acting as a reservoir of medication and a means to connect the patient to the Internet portal. These devices had the ability to store the medication and would remind the patient at the appropriate time to take their medication. Any instances of non-adherence would be recorded by these devices 46 Remote Healthcare Monitoring and Assessment

• Doctor: a web based interface (Figure 3.5(b)) to support the prescribing of medication and assessment of patient adherence. This interface provides a means for the medication regime of the patient to be entered onto the system. • Pharmacist: a software interface (Figure 3.5(c)) to support the ﬁlling of medication containers to be used by the patient’s medication device according to their prescribed medication regimen. This system has the ability to retrieve the information entered onto the system by the Doctor. • Caregiver: this software interface (Figure 3.5(d)) provides a means to allow the patient’s adherence to their medication regimen to be monitored in real time and, in instances of non-adherence, will raise the attention of the care giving staff.

(a) Mobile medication device (b) Doctor’s Interface

Figure 3.5: Stakeholder interface for the MEDICATE system. Remote Healthcare Monitoring and Assessment 47

Within this scenario we have demonstrated how the basic concepts of utilising modern day communications have supported the integration of a number of stakeholders in the supply-to- intake chain of medication. Evaluation of this system has demonstrated its usefulness in terms of supporting a patient’s adherence to medication. Nevertheless, it has also highlighted the importance which should be given to the integration of such systems into existing practice. The natural evolution of this system has involved the deployment of the concepts of medication management onto mobile phone based platforms [95]. These facilitate a similar means of providing management and of monitoring of a patient’s intake of medication, in addition to maintaining the link with healthcare professionals. An additional extension to the system would involve integration with context aware services, where the means by which the reminder was offered to the patient would be dependent upon their current activity.

3.5.4 Conclusions

Overall, within this review we have shown how technology can be used to establish a number of care paradigms, all of which aim to improve the delivery of care provided to the patient and to overcome geographical boundaries. In some cases existing technological infrastruc- tures may be used and in others new forms of technology are required. Prior to widespread deployment, a number of challenges still remain. These challenges are only partly related to the technology. Other challenges are more related to organizational issues and issues relating to who is actually going to pay for the technology and its associated services. Nevertheless, it is widely accepted that there are beneﬁts to be accrued, hence it is anticipated that further effort will be directed towards these areas in future years. 48 Part 4

Wearable Systems in Nursing Home Care: Prototyping Experience

Citation: Mikael Drugge, Josef Hallberg, Peter Parnes, and Kåre Synnes, "Wearable Systems in Nursing Home Care: Prototyping Experi- ence",InJournal of IEEE Pervasive Computing, vol. 5, no. 1, pages 86–91, January–March 2006.

Contributions: This paper was achieved together with Mikael Drugge. Mikael was the initiator of this work and is the main author of this paper. Though the work leading up to this paper, such as interviews and tests with elderly- care personnel, was shared equally, Mikael did most of the writing. I did, however, write the section about the Wizard of Oz method used in our tests.

Wearable Systems in Nursing Home Care: Prototyping Experience 51

Wearable Systems in Nursing Home Care: Prototyping Experience

Mikael Drugge, Josef Hallberg, Peter Parnes, and Kåre Synnes Department of Computer Science and Electrical Engineering Luleå University of Technology SE–971 87 Luleå, Sweden {Mikael.Drugge, Josef.Hallberg, Peter.Parnes, Kare.Synnes}@ltu.se

4.1 Introduction

Medical workers at nursing homes spend much time on communication to get the right information to the right person at the right time. This communication is a prerequisite for proper patient care. Delays cause stress, discomfort, and dissatisfaction among caretakers and patients as well as possible detrimental health consequences for patients. We believe pervasive computing technologies can improve this situation by speeding communication and documenting care more effectively. In fact, pervasive computing is a promising solution to many problems that medical workers face, and today it’s increasingly practical. Yet actual deployment is still in its infancy. Deploying prototypes that solve speciﬁc problems can help medical staff see its beneﬁts. In- volving them early in the design process also helps ensure that the right problems are being solved and the solutions will be accepted. We decided to try rapid prototyping in a real nursing home. We set a tight four-week deadline for ourselves and began work to build a testable prototype with two half-time developers. This gave us one person-month to develop a useful prototype to investigate research questions including:

• What methodologies are useful for prototyping pervasive computing systems? • How do we engage end users in prototype design and interactions? • Can we rapidly deploy prototypes built from existing technology in real settings? • How can we translate conceptual solutions to functional prototypes?

We worked with the Lighthouse, a local nursing home that provides short-term residential care in apartments. Self-sufﬁcient elderly who’ve been set back by accidents or illnesses can receive the support they need to recuperate. With up to 40 guests, the Lighthouse sees a continuous stream of patients. Although busy, it’s small enough for us to easily deploy and study prototypes in real situations. 52 Wearable Systems in Nursing Home Care: Prototyping Experience

4.2 Scoping the Project

We first had to identify actual problems nurses face in everyday work, determine which could be solved within our research’s scope, and identify which would bring the most gain. Al- though we had reports on typical problems, we decided a field study would give us first-hand experience with their work. This was accomplished by a "quick and dirty" ethnographical study [63], where we accompanied a group of four nurses for a day and observed the professional tasks they perform. During this study, we observed scenarios such as drawing blood samples and administer- ing pain medication. A consistent theme for many tasks was the difficulty in getting necessary patient information at the point of care. The patient charts contain the most important information, but they are only accessible from the nurses’ office computers. Nurses typically must walk hundreds of meters and change floors to attend patients in their apartments. Going back to the office computers takes time, so the nurses need a system which supports retrieving such information in advance and updating the patient charts upon returning to the office. The nurses currently keep updates in short-term memory or handwritten notes. When nurses need more informal information, they must be able to contact the person who previously cared for the patient. Typically, the Lighthouse nurses used mobile phones for this purpose. When the phone calls reached the previous caretaker at all, and often they didn’t, they usually interrupted the recipient. Likewise, other people calling the Lighthouse nurses interrupted their patient care, increasing stress and discomfort for both the nurse and patient. We also found mobile phones lacking multimodal features, supporting only voice, while the situation itself required video to convey certain information. For example, rather than having a patient point out pains directly to a physiotherapist, the nurses must relay this information over the phone, thereby losing the subtle details that body language can reveal. At the end of the day, we summarized the problems we encountered, both in scenario form and as a list of specific items, including

• communication, • information dissemination, • access to patient charts, and • organizational issues.

A few days later, we went back to the Lighthouse for a meeting with the nurses to validate our findings and ensure that the identified problems were real. This helped us decide which were the most important. Access to patient charts involves strict privacy and security considerations, and remote accessing requires detailed analysis and evaluation of security infrastructure. So we simulated this information for our prototype. Organizational issues such as staffing, budgets, and work schedules are economic and political issues that we won’t discuss further. Wearable Systems in Nursing Home Care: Prototyping Experience 53

We could address the communication and information dissemination issues among the personnel within the limited scope of project. These are closely related, since communication is a way of having the right information for the right person at the right time. We discussed these issues with the nurses and came to a joint conclusion regarding the research prototype’s focus. The consensus was that it should support easier communication among the personnel and also function as a documentation tool for informal notes. It should be mobile and allow for access from anywhere in the building, be less intrusive than a mobile phone, and employ a highly streamlined interface to avoid taking focus from the patients.

4.3 Paper Prototyping

Hardware inventions to embody and run pervasive applications are often wearable or highly portable. Research concepts often rely heavily on unique hardware and require working prototype to test and illustrate the operational concepts. Producing these prototypes can be prohibitively expensive and time consuming. You can simplify hardware prototyping by using modular approaches – for example, the Smart-Its project [142] operates this way. Yet such prototyping remains focused on hardware technology, which runs the risk of distracting from function and usability. Traditional HCI researchers have used paper prototyping to good effect [119]. This simply involves drawing user interface components on paper, making it easy to alter designs and fix flaws early. Not everyone can use design software for prototyping user interfaces, but everyone knows how to draw sketches with a pen. So paper prototyping allows end users to become part of the design process early. Paper prototyping is so artificial that it can remove the focus on technology. Instead, participants can focus on the product’s underlying concept and usability. This works, however, because of the fixed and rigid desktop paradigm with its graphical presentation space that can be adequately represented on a sheet of paper. Paper prototyping in pervasive computing applications involves additional challenges. The multimodal interaction in such environments supports more complex scenarios than desktop GUIs accommodate. This freedom can mean the paper itself imposes restrictions on what can be done – for example, not having access to large sheets of paper can prevent participants from envisioning whiteboards, while lacking small sheets can discourage them from thinking of handheld devices.

4.3.1 Paper, Pen, and Plastic

Following our requirements study with the nurses, we arranged a meeting to try our technique. We informed them that we needed their input and feedback to design a prototype that would be useful to them and that we would employ paper prototyping. We prepared scripted scenarios from the data we’d collected on typical nursing tasks, such as visiting and examining patients, informing physiotherapists, taking blood samples, and making rounds with physicians. We let them role-play the scenarios and discuss various solutions to problems they encountered. One of the nurses played herself in work situations, while two others played patient roles. Each patient player received a brief description of the scenario. 54 Wearable Systems in Nursing Home Care: Prototyping Experience

In addition to the scenario, we had text cards representing typical phone calls. We emu- lated a context-aware service that could determine whether the nurse was currently occupied and could then choose between presenting phone calls directly or taking a message. For calls that weren’t urgent, we displayed a simulated text message on a transparent piece of plastic. We then either set it in front of the nurse to view in private or placed it on the device the nurse carried. Presenting these messages randomly during the role play allowed the nurses to decide what visualization form they preferred. After completing the scenario, we talked with the nurses about their experience. They appreciated being able to avoid interruptions from non-acute calls through simple mechanisms such as screening incoming messages. They liked having text messages presented because it let them read without interrupting what they were doing. They found it useful to have a patient’s information available in advance or upon contact because it let them better prepare for encounters with special patients. For example, if a patient posed difﬁculties in taking blood samples, the nurses could add extra test tubes to their medical kits. The nurses also liked having the live video to directly show, for example, a patient’s shoulder fracture. They thought this would save them time that they could spend on more important tasks. In general, the paper prototyping session made the prototype’s beneﬁts clear without having to deploy fully functional hardware and software.

4.3.2 Paper Prototyping Beneﬁts

The beneﬁts of early paper prototyping over an online, functional software-hardware prototype became clear at the end of the meeting. We brought some hardware just to show the nurses what is available with today’s technology. Immediately, we noticed their focus shifting away from usefulness to technical details regarding the software and hardware. They ques- tioned font sizes ("too small for me to read"), commented on video quality ("better than the one we saw on another project"), and expressed astonishment over features ("so I could even write my emails on this"). As their focus scattered, they often addressed details irrelevant to the tasks they would need to perform. Importantly, while interacting with the computer, we noticed the nurses started to think in terms of traditional user interface widgets, such as buttons, menus, and keyboards, and to restrict themselves to the kind of interactions typical of desktop PC applications. They also appeared more dejected and hesitant to suggest improvements, and they expressed slightly negative comments and questions such as, "We’ll need to take courses to understand this. You’ll arrange that for us, right?" In general, the nurses shifted their entire focus, assuming now that only minor changes in technical details were possible. Clearly, such restrictions in the design space should not dominate the early stages of prototyping. We concluded that paper prototyping in the initial development stages makes participants focus more on a product’s concept and actual usefulness rather than letting technology con- strain their thinking and dictate what is allowed. Furthermore, unlike technology, paper does not restrict the design space, allowing participants to think beyond the inherent limitations of current software and hardware. We see the same beneﬁts for pervasive computing research that it exhibits for traditional HCI in desktop computing. Taking merely one week in preparation, execution, and analysis, we deem this time well spent. Wearable Systems in Nursing Home Care: Prototyping Experience 55

4.4 Moving to Multimodal Devices

With the newfound design considerations from the paper prototyping session in mind, we went on to see what research technologies could help in realizing a multimodal prototype. Our research group’s background is based in real-time audio and video communication over the Internet, and we have extensive experience in desktop e-meeting tools. We’ve also ven- tured into the pervasive computing field, investigating mobile and ubiquitous applications of the technology. Beyond technical barriers, we’ve found that the end users’ concerns and preconceptions often determine whether new systems are adopted. For this application, the prototype had to be mobile and non-intrusive, which fits well within the field of wearable computing. We also saw healthcare applications requiring special considerations, as illustrated in one nurse’s opinion: "It was not to sit in front of computers I chose this profession". The prototype had to avoid the negative emotions that time-stealing and crashprone desktop computers currently cause. An important way to hide technology and streamline the interaction is to use context and situation awareness, automating the information presented to the nurses according to the current activity and workload. Some of these concerns are major research problems in themselves, which meant that our online prototype couldn’t realize all concepts within a reasonable time frame. However, by employing a Wizard of Oz approach to the user interface [31], we could still demonstrate the functionality and get the nurses’ opinions. Knowing how, or whether, the nurses used certain functions let us select what areas to put the most effort on in future prototypes.

4.4.1 Wearable Prototype

The next step was to build an online prototype with live software and hardware. We wanted the prototype ready within a week from the paper prototyping session. This gave us no time to build customized hardware, meaning we had to assemble our prototype from off-the-shelf products. As a wearable computer, we chose a Sony Vaio U70P, a notebook computer with a 1-GHz Pentium mobile processor and 512 Mbytes of memory. With dimensions of 16.7 x 11 x 2.8 cm and weighing 550 g, it can be worn easily by strapping it to a belt, which the nurses deemed suitable during our last meeting. Because the nurses liked viewing information in private, we added a head-mounted display as an alternative to looking at the U70P. We chose the semitransparent monocular M2 Personal Viewer with full-color graphics in 800 x 600 resolution, even though it requires a half-kilogram battery. Since the prototype demonstrates concepts rather than a ﬁnal product, we deemed the quality of graphics to be more important than the additional weight at this stage. Figure 4.1 shows the wearable computer and display.

4.4.2 Communication Application

We chose Marratech (www.marratech.com) e-meeting application software because it ﬁts the communication needs of the envisioned prototype. Marratech is a commercial product based 56 Wearable Systems in Nursing Home Care: Prototyping Experience

Figure 4.1: Wearable computer outﬁt: battery pack and VGA converter for the head-mounted display (left), Sony U70P computer (lower center), Bluetooth headset (upper center), and M2 head-mounted display (right). on earlier research in our group [107]. It allows for audio and video group communication, together with text chat and a shared whiteboard. Connecting the wearable computer to an e- meeting over a wireless network lets the nurse instantly contact other participants and become aware of their locations. The application also lets a nurse make phone calls, which can become part of the e-meeting. This allows persons not yet using the software to be included, easing its deployment. Figure 4.2 shows the wearable computer running the Marratech Pro application, with video streams provided by Web cameras. The nurse can use the camera to convey live video to physiotherapists or others to aid diagnoses.

4.4.3 Wizard of Oz Testing

We wanted to show a simple and automatic system to the nurses, so we decided on a Wizard of Oz experiment to simulate the context-aware and situation-aware system components. The wizard retrieved information, processed interrupting phone calls, simpliﬁed communication with others, and minimized the interaction needed with the wearable computer. Each morning the nurses make several calls for information regarding their patients that day. In our prototype, the wizard collects this information and displays it in the shared whiteboard, thus shortening the time nurses need to set their daily schedule. With access to pa- Wearable Systems in Nursing Home Care: Prototyping Experience 57

Figure 4.2: The Sony Vaio U70P running the Marratech Pro e-meeting application. tients’ charts, historical notes, and other nurses’ locations and schedules, the wizard can post appropriate information throughout the day. If a nurse is attending to a patient or is otherwise busy, the wizard intercepts all phone calls. It sends only urgent calls directly to the nurse and either records the rest or posts them as a chat room message to read when there is time. This substantially reduces interruptions during patient encounters. When nurses wish to reach someone, the wizard can invite that person into the e-meeting session, thus allowing richer communication than normal phones. To further simplify the nurses’ work, we let the wizard act as a speech recognizer. Nurses can take informal notes about certain patients for insertion into their chart. They can also request information to be read out loud, which further simpliﬁes the interaction with the wearable computer. In addition, the nurses can use voice to insert new tasks into their schedules. Most of the wizard’s functions can be realized with today’s technology, such as RFID tags, sensors, and speech recognition capabilities.

4.4.4 Feedback From the Nurses

We brought our wearable prototype to the Lighthouse to test it for a day, starting with audio- only communication. We equipped one nurse with the wearable computer and a Bluetooth headset, while another nurse in another room used a laptop. We connected both devices to the same e-meeting, which effectively mimicked mobile phones but reduced interruptions and offered higher audio quality. We also allowed review of a ﬁctitious patient history and chart to demonstrate the capability. 58 Wearable Systems in Nursing Home Care: Prototyping Experience

Figure 4.3: A nurse wearing a computer and head-mounted display while attending a patient.

Next, we introduced video communication. As one e-meeting participant walked the corridors, the nurses expressed their fascination with instantly seeing where their colleagues were. This increased group awareness seemed beneficial, especially for finding the right person at the right time. Initially, the nurse sent video via a handheld camera and used the notebook’s display to view other participants. Because these tasks encumber the nurse’s hands, we also let them test the head-mounted display and camera. After a brief time, about 15 minutes, they accustomed themselves to the novel display and learned to focus on the display or the real world as needed. Soon the nurses could easily perform routine patient examinations while conveying video to other medical workers. This aided patient diagnosis and treatment deliberations. Figure 4.3 shows a patient encounter. Although noting the added weight, the nurses remained focused on the design concept being demonstrated and its benefits. Thus, on the basis of results from paper prototyping and a wearable prototype built from off-the-shelf components, we successfully demonstrated the envisioned benefits and gained user acceptance for an assistive device in this environment. By starting with the basic functions and gradually adding more, we avoided intimidating the users with the amount of hardware and cables.

4.5 Final Remarks

About a month after the online study, we revisited the Lighthouse to discuss the prototyping in hindsight. Despite the time that had passed, the nurses still deemed the prototype useful and appropriate. We see this as confirming that the process yielded usable results and Wearable Systems in Nursing Home Care: Prototyping Experience 59 that ethnographical studies and paper prototyping can be effective in pervasive computing research. Ethnographical study provides valuable first-hand insight on how work is performed and gains the confidence of the user community. Paper prototyping offers a cheap and easy way to get quick feedback on what constitutes a good or bad solution. Furthermore, people not in the pervasive computing research community still consider much of what is possible with today’s technology as science fiction. We found it difficult to get those people to envision what’s possible, and we had to find ways of freeing them from traditional PC interface ideas, while not imposing our ideas on how things should be done. Paper prototyping can aid this to some degree, and joint discussions of paper results encourage fuller exploration of the design space. It also moves the participants’ focus from technology to usability and function. One major challenge we found concerning paper prototyping is to communicate what is realizable without constraining participants from exploring the whole design space spectrum. We think the best solution is to introduce concepts in a brainstorming session before the paper prototyping. You can add the wildest ideas to the hardware research agenda and use ideas that can be realized with current hardware technology in the prototype. This should allow for more freedom of thought in the subsequent paper-prototyping stage while constraining the overall process to prototypes possible given the available time, budget, and technology. Realizing the paper prototype in functional hardware reveals differences between reality and visions that can require compromises. The Wizard of Oz approach allows for emulation of functionality not immediately realizable. However, if the prototype is meant to be deployed for longer term studies, the researcher should be sure this functionality is realizable within the envelope of current technology. Finally, rapid prototyping with end-user involvement from the start has a value in itself. The nurses we worked with spontaneously expressed enthusiasm for our project due to its fast pace. They felt that something was happening and that their input was valued. As opposed to other projects where meetings are half a year apart, rapid prototyping let them see progress from week to week.

4.6 Acknowledgments

We wish to acknowledge funding by the Centre for Distance-spanning Healthcare at Luleå University of Technology. We also express our thanks to the nurses and medical workers at the Lighthouse (in Swedish, "Fyrens korttidsboende") in Luleå, Sweden, for their participa- tion. We also thank the reviewers for valuable guidance. 60 Part 5

Supporting Automatic Media Resource Selection Using Context-Awareness

Citation: Johan Kristiansson, Josef Hallberg, Sara Svensson, Kåre Synnes, Pe- ter Parnes, "Supporting Automatic Media Resource Selection Using Context-Awareness",InProceedings of the 3:rd International Con- ference on Advances in Mobile Multimedia (MoMM2005), pages 271– 282, Kuala Lumpur, Malaysia, September 2005.

Contributions: This paper was accomplished while working closely together with Jo- han Kristiansson and Sara Svensson. Johan was the one who initialized the work and is the main author of the paper, though we all contributed equally in the development of the prototype. In the end we both contributed the same amount of work in producing the paper where my main contribution was the introduction of indexes which are used for comparing resources to each other and selecting the most suitable resource. I also undertook most of the writing in the evaluation section, most of the related work and the discussion. Johan and Sara were mostly responsible for the algorithm, though it was reﬁned by me. Jo- han also contributed signiﬁcantly to the introduction and the evaluation section. Sara’s main contribution was the section about media resource management.

Supporting Automatic Media Resource Selection Using Context-Awareness 63

Supporting Automatic Media Resource Selection Using Context-Awareness

Johan Kristiansson, Josef Hallberg, Sara Svensson, Kåre Synnes, Peter Parnes Department of Computer Science and Electrical Engineering Luleå University of Technology SE–971 87 Luleå, Sweden {Johan.Kristiansson, Josef.Hallberg, Sara.Svensson, Kare.Synnes, Peter.Parnes}@ltu.se

Abstract

In a world of ubiquitous media resources, such as cameras, displays, microphones, provided by a wide variety of multimedia systems, there is a need for automatic resource selection which seamlessly utilises the best available communication tool from a user perspective. This paper therefore presents an algorithm which uses context-awareness to support automatic media resource selection. The algorithm takes advantage of an abstract classiﬁcation of privacy, quality, and cost in order to compare media resources to user’s preferences. As a proof of concept implementation, the algorithm has been incorporated into an e-meeting application called Marratech.

5.1 Introduction

As users’ requirements on mobility grow and distributed collaboration becomes increasingly pervasive, the need grows for people to communicate and retain group awareness in mobile settings. Current research trends in networking and multimedia envision ubiquitous multimedia communication where users can seamlessly meet and communicate anytime, anywhere, and from any device. Imagine a world full of communication equipment and softwares, such as cameras, microphones, instant messengers, and e-meeting clients 1, which are scattered throughout the environment and are automatically conﬁgured and utilised whenever the user wants to communicate. This scenario raises new challenges, which must be addressed before the vision can become a reality. Today, users are required to manage a wide variety of communication tools in order to communicate with other users. These tools typically force the user to manually set up communication links, for example entering phone numbers, logging in to e-meeting servers, and conﬁguring cameras. In addition, many tools are inherently incompatible, which requires all communication parties to have access to the same set of tools in order to be reachable by

1The term "e-meeting" denotes a group teleconferencing session that can include video, audio and chat among other media. Rather than requiring a dedicated meeting room, e-meetings can take place from the user’s desktop and be used for either formal or informal communication. 64 Supporting Automatic Media Resource Selection Using Context-Awareness each other. The conﬁguration task becomes even more complicated and time-consuming as the need for ubiquitous communication spreads and the diversity of available communication tools increases. Ultimately, the best available communication tool should automatically be utilised trans- parently to other users and independently of which tools they are currently using. Re- search has been conducted to provide interoperability [108, 134] and support for mobility [79, 130, 146] to allow use of different tools. However, how to enable a system to automatically choose the most beneﬁcial tool for the user is a problem which so far has received less attention because of its complexity. This paper addresses this problem by decomposing it into more manageable parts, which can be solved independently, in order to provide a deeper understanding of the underlying problems. The paper proposes an algorithm which connects these parts together and uses context-awareness to decide which communication tool that is best for the user. As a proof of concept, the paper demonstrates how the proposed algorithm can be used to automatically switch between communication tools when using a commercially available e-meeting software. By using the prototype, users can participate in an e-meeting session and the system automatically invites an IP-telephony service, which includes a telephone or a mobile phone to the session, as a complement when a desktop e-meeting client does not provide good enough privacy. Similarly, the system can automatically invite an IP-telephony service if the currently used network connection does not provide good enough performance to support audio. The rest of the paper is organized as follows. Section 5.2 describes related work and the contribution of this paper. Section 5.3 introduces a conceptual communication model and discusses issues that must be considered when selecting resource. Section 5.4 outlines the proposed algorithm and discusses two issues related to its usage. Finally, in section 5.5 the proof of concept prototype is described followed by a discussion and future work in section 5.6.

5.2 Related Work

Context-awareness is a widespread approach to enable systems to provide users with an increased service value. The area of interest to this paper is context-aware communication as defined by [126]. Early work in this field include the ActiveBadge system [148], which tried to improve channel selection through autonomous routing of phone calls to the phone nearest to the user. In contrast, this paper focuses on autonomous routing of media streams to the most appropriate communication resource from a user’s perspective. Similar to the architecture proposed in this paper, the Aura architecture [137, 147] tries to utilise resources in the user’s environment to give the user access to desired tools, while simplifying the configuration tasks needed by the user. In contrast to the algorithm proposed in this paper, Aura does not consider the possibility of several resources with the same capabilities. Hence, Aura does not provide any methods for comparing resources in order select the resource that best satisfy a user’s needs. Neither does Aura take other aspects, such as cost, into account when selecting a resource. Supporting Automatic Media Resource Selection Using Context-Awareness 65

The Aura architecture identiﬁes simple tasks performed by the user, such as editing text, and then migrate the whole task to a different resource as the user change location, for instance moving a text from one device to another while changing text editor if needed. This means that Aura does not take advantage of the possibility to split up certain tasks and then distribute these tasks to different resources that better meet the user needs. One such task is the communication task which can be split up into sending and receiving of different media. The algorithm proposed in this paper treats these parts as media resources that are selected separately which makes it possible to distribute the tasks onto different devices. Architectures which focus on network mobility [79, 130, 146] aim at providing support for location updates and for preserving established connections when changing terminals or moving across networks. This paper does not look into network mobility in this sense, but rather on how to provide support for automatic selection of appropriate terminals to switch to. The main contribution of this paper is an algorithm which decomposes this selection problem into more manageable parts, thus making it possible for mobility architectures to become more autonomous.

5.3 Media Resource Management

This section starts with presenting a conceptual communication model which is used throughout this paper. It then goes on with describing the different issues which needs to be considered in order to enable a system to automatically make a choice which suits the user. In particular, the section considers the problems of valuing and comparing resources and proposes possible solutions to them.

5.3.1 Communication Model

Figure 5.1 illustrates the used conceptual model showing two communication abstractions, media source and media sink. A media source is an abstraction over a media-capturing hardware device, such as a camera or microphone, and the software that generates the media stream. After being generated, the media stream is distributed to the recipients via a network infrastructure. When the media stream has been received and processed by the recipients, a media sink presents the media stream to the user. A media sink is an abstraction of a media- rendering hardware device such a loudspeaker or a display and its accompanying software, which is used by the device to send or receive the media stream. The word media resource is used to denote either a media sink or a media source in the rest of this paper. For the sake of simplicity, this paper introduces the term media resource type to be able to differentiate between sinks and sources of a particular media. For example, all media sinks which are associated with microphones are of the type audio sink, and all media sources associated with cameras are of the type video source. This makes it possible to conceptually handle situations when a user only wants to either send or receive a particular media. Me- dia resources which are contained within the same unit are deﬁned as interdependent media resources. For example, a phone has two interdependent media resources, one of the type 66 Supporting Automatic Media Resource Selection Using Context-Awareness

Media source Selected media Media sink resources Infrastructure Media sink Media streams Media to/from other gateway users Media sink Media source

Figure 5.1: A conceptual communication model. audio sink and one of the type audio source, and an e-meeting client might have even more interdependent media resources. One of the advantages of using the proposed abstractions is that each media resource can be viewed and handled as an isolated unit, thereby enabling computer-mediated communication to be enriched and improved by letting users freely combine different media resources. To provide such functionality, the system needs to be able to automatically detect and select the best available media resources, and ensure compatibility between senders and receivers. For example to allow a mobile phone to be used as an audio source in an e-meeting session, a SIP-gateway can be used to convert Public Switched Telephone Network trafﬁc to IP trafﬁc. The problem of ensuring compatibility is not addressed in this paper.

5.3.2 Media Resource Selection Problems

One problem with automatic media resource selection is what level of autonomy should be used. As pointed out in [126], autonomy needs to be carefully balanced in order to provide the desired effect. It is crucial not to remove too much of the user’s sense of control as well as to avoid switching to a media resource which the user does not want to use. These types of system failures will most likely not be tolerated. However, at the same time, involving the user in too many of the decisions will burden the user and counteract the requirement of minimal effort. When it comes to autonomously deciding which media resource to switch to, there are several problems which need to be considered; when to look for a better resource, when to switch, and when a speciﬁc media should be used. Solving these problems in a good way requires determination of complex causalities and access to a wide variety of contexts, e.g. network performance of current resource, improvement achieved if changing, a user’s preferences, and so on. User studies need to be conducted in order to determine these causalities and which contexts to use. To clarify what information that is relevant to base the decision upon, two concepts need to be more clearly deﬁned, namely best resource and available resource. This can be done by discussing the following two design issues which are related to the two concepts respectively:

• How can media resources be compared? • Which media resources should be considered available in a given situation? Supporting Automatic Media Resource Selection Using Context-Awareness 67

The rest of this section discusses these two issues and proposes possible solutions.

Comparing Media Resources

In order for the system to be able to decide between media resources, they have to be com- parable in some way. Through this comparison, it must be possible to value media resources in a way that concur with the user’s preferences, thus enabling the system to act according to those preferences. One dimension which is interesting from a user point of view is of course quality. The user will most likely want to use the better one of two microphones in a room. When considering media resources, privacy will also play a significant role in what the user prefers. Parameters such as the number of people in the room, what part of the room that is captured by the camera and how publicly the display is located will most likely affect the user preference. In a future with an abundance of media resources, it is quite likely that all resources can not be used for free. This introduces a third interesting dimension, namely cost which can involve both monetary and non-monetary values. For example, the cost involved could be to pay a given amount of money per byte or allowing a newsletter or advertisement to be sent to the user. These types of costs are difficult to determine, although for different reasons. The non-monetary costs will have different values to different users, making valuing of resources in a way which concurs with user preferences complicated. Monetary costs are often difficult to decide since the actual cost in many cases is connected to how much data is transferred or how long time the resource is used. A classical solution when valuing an object is to use some form of classification. To use one quality, one cost, and one privacy classification level to prioritize media resources is thus one way to approach the problem of comparing media resources. Besides giving a foundation for comparison, the use of classification levels will also make it easier for users to convey their preferences in different situations, e.g. "best possible quality, least possible cost, and at least privacy level seven". To use classification also provides researchers and developers with an abstraction which separates the issue of valuing media resources from the rest of the selection process. In order for classification levels to function correctly, the system should be able to reflect a user’s opinion of what each level should entail. This implies that users need to have the possibility to influence how the classification levels are calculated. For example, one user might consider it unacceptable (low privacy level) to use a microphone when there are other people in the room while another user might tolerate it (higher privacy level). Another issue is how the different classifications should be weighed in order to take importance into account. This calculation also needs to take the user preferences into account. For example a user might consider bandwidth more important than cost in a certain situation and the opposite in another situation. A different user might always prioritize cost before both privacy and quality. As many environments are dynamic, a media resource’s classification values will change over time. When considering the quality level, network specific context such as packet- loss rate and available bandwidth makes dynamic quality classification a necessity. Also in the case of privacy, the level need to be dynamically recalculated since a resource could be 68 Supporting Automatic Media Resource Selection Using Context-Awareness considered to have a lower level of privacy if there are other people in the room, than if the user is alone.

Media Resource Availability

Assigning values to media resources allows for comparison between them, but in an environment which provides a large amount of media resources, it is neither efﬁcient nor preferable to always consider every resource as a possible choice. This brings forward the question of what characterizes an available resource in a certain situation. Naturally, only devices which are not currently busy with other users and which provides the wanted media resource type are relevant, but there are other things to consider as well. For example, when the user wants to send audio it would not be beneﬁcial to the user if the system decided to use a microphone in a building other than where the user is located. This can be solved by using the principle of locality [73]. Locality means that the user only is allowed to use devices which are nearby the user, for example in the same room. Locality makes sense not only for reasons of usefulness but also when considering privacy. Without the principle of locality, cameras and other media sources could be used as remote surveillance devices, thereby introducing a large risk of violating other people’s privacy. Locality also ensures that media sinks are not used without the receiver’s permission, thereby protecting the receiver from unwanted interruptions. Two issues which need to be considered when applying the principle of locality is what is to be considered as close enough and what should happen when there is not any available resource within this area. Even if the principle of locality is useful, it is not necessarily true that a user should have access to all nearby resources. This further narrows the number of relevant resources to those nearby the user which are not busy and which the user has permission to access. The presence of interdependent media resources also affects which resources are relevant. If a user uses an interdependent resource, no other user should be able to use the other resources which the used resource is interdependent with. For example, if a user is utilizing the microphone in a mobile-phone, that user should be the only user able to use the speaker in the same device. This could be seen as a special case of media resources being busy, where all resources which are interdependent with the used one are considered to be occupied.

5.4 Automatic Media Resource Selection

In order for a system to deal with the problems discussed in subsection 5.3.2 and choose a suitable media resource, it needs to access and use a situation’s context, i.e. it has to be context-aware. A system is said to be context-aware "if it uses context to provide relevant information and/or services to the user, where relevancy depends on the user’s task" [33]. Context is in this case any information relevant for the system to make a suitable choice, e.g. user preferences and information about available media resources. Note that this paper does not focus on context acquisition or accompanying problems like context accuracy or conﬁ- dence. In this section an algorithm for automatic media resource selection is presented. This Supporting Automatic Media Resource Selection Using Context-Awareness 69

Algorithm 1 Media Resource Selection Algorithm Input: A media resource type, type, e.g. audio sink or video source. Output: The best media resource, rselected , for type. 1: rpreferred ⇐ null 2: if isNeeded( type ) then ⇐ ∀ ∈ 3: R all media resources r of type s.t. r R isAvailable( r ) Q( r ) ≥ QMIN P( r ) ≥ PMIN C( r ) ≤ CMAX 4: for all r ∈ R do 5: if credit( r ) > credit( rpreferred ) then 6: rpreferred ⇐ r 7: end if 8: end for 9: end if 10: rselected ⇐ notify( rpreferred ) algorithm breaks down the problem of choosing the best media resource into several abstract functions which can be implemented separately using context-awareness. This section also discusses how the algorithm should gain access to contexts and what should happen once a decision has been made.

5.4.1 A Media Resource Selection Algorithm

The purpose of the Media Resource Selection Algorithm (MRSA) is to select the best available media resource in accordance with user preferences. To do this, the algorithm is based on the discussion in subsection 5.3.2 and it uses classiﬁcation of quality, privacy, and cost to model media resource values. The algorithm is presented in algorithm 1. The algorithm returns rselected , which designates the selected media resource of the inputted type, type. Throughout the algorithm, r preferred is used to denote the media resource which is the currently preferred choice. Further, the variables P MIN and QMIN signify the minimum values of privacy and quality, and CMAX the maximum cost value that a user favours. The algorithm decomposes the selection problem into seven abstract functions which need to use context-awareness in order to function properly. These are in order of appearance; isNeeded, isAvailable, Q, P, C, credit,andnotify. The boolean function isAvailable(mediaresource) checks if a given media resource is available for use. If the function follows the solution proposed in the Media Resource Availability part of subsection 5.3.2, the function should return true if the given resource is nearby, not busy, and if the user has permission to use that particular resource. Each of the functions isNeeded and notifyencapsulates different aspects of the autonomy problems discussed in subsection 5.3.2. The boolean function isNeeded(type) checks if the given media resource type is needed, i.e. if the user wants to communicate using that media resource type. For example, if a user sends audio in an e-meeting session, the media resource type audio source is considered needed for the user who sends the audio stream, and the media resource type audio sink is considered needed for all users who receive the audio 70 Supporting Automatic Media Resource Selection Using Context-Awareness

Select and Media configure Sensors Resource media resource Query Manager

Context storage infrastructure Context MRSA

Figure 5.2: Algorithm Usage stream. To decide when the user wants to start or stop using a specific media resource type, more sophisticated reasoning about contexts is needed compared to when the media resource type is continuously used. The function notify(mediaresource) is used to decide if the user wants to switch to the given media resource or not based on context. This could for example include asking the user if a switch should be performed. The argument can either be the currently used media resource, a new and better media resource, or null. The function then decides between con- tinuing to use the currently used media resource if that is an option, switching to the new media resource, or not using any media resource of this type at all. The function returns the selected media resource. The Q, P,andC functions all take a media resource and calculates the classification level of quality, privacy, and cost respectively. The function credit(mediaresource) weighs the different classification dimensions together according to user preferences in order to compare them. All of the abstract functions in the algorithm need to take a large amount of context parameters into consideration and weigh them together in order to achieve user satisfaction and are therefore difficult to implement, which is illustrated through the discussion in section 5.3.2.

5.4.2 Using MRSA

When considering deploying and using the MRSA, there are two issues which need to be discussed. These are; how should the algorithm get access to necessary contexts that are needed to make decisions and how the decisions should be executed. Regarding the ﬁrst issue, a context storage infrastructure with query-possibilities must exist. It must also be possible to add and update contexts to the infrastructure. One approach for developing a context storage infrastructure is to use a context-awareness platform. Schilit was one of the ﬁrst to propose an architecture for this purpose [128]. Since then several context-awareness platforms have been developed, each with different characteristics, e.g. Context Toolkit [34] and Context Fabric [60]. Current research moves toward decentralized solutions for privacy and scalability reasons [60]. Supporting Automatic Media Resource Selection Using Context-Awareness 71

When using the algorithm in an existing communication system, the algorithm normally resides in an external or internal management component responsible for conﬁguring and utilizing appropriate media resources. The management component, the Personal Media Re- source Manager, could be implemented as part of a multimedia system. An overview of how the algorithm can be used by a Personal Media Resource Manager together with a context storage infrastructure is illustrated in ﬁgure 5.2.

5.5 Evaluation

A proof of concept implementation was built by incorporating the architecture with a commercially available e-meeting system provided by Marratech [80]. Marratech is an e-meeting client which provides tools for synchronous interaction including audio, video, chat, and a shared white-board. The client connects to a media gateway called Marratech Manager in order to access and participate in e-meetings. This system was mainly chosen because of the SIP gateway which is built in to the manager, and compatibility with other prototypes developed by the authors. The rest of this section describes the implementation and two test cases in which the prototype has been used.

5.5.1 Implementation

The prototype is implemented in Java JDK 1.4 in order to make it easy to integrate into the Marratech source code. The implementation uses the described algorithm, with the exception that cost has not been taken into account, and is organized as figure 5.2. The current implementation does not separate sinks and sources in phones, hence when either the microphone or the speaker in a phone is selected the other is automatically used. A sensor simulator was used to generate two contexts, each of which significantly changes a user’s situation, namely location and network performance. The context storage infrastructure models media resources, environments, and users as Java objects. Each environmental object models a room and keeps information regarding which media resources and which users reside within the room. The user object contains relevant contexts such as active media resources, user’s location and a link to the appropriate environment object, and which media resource types the user prefers. By letting the context storage infrastructure model relationships between users, resources, and locations in this manner, e.g. media resources are linked to locations, the infrastructure provides a way to discover resources. Finally, the media resource object pertain contexts for calculating quality and privacy indexes, as well as a SIP address. A communication protocol was also designed to enable the used sensors to add context to the infrastructure. The Marratech client reports if audio is being used to the context storage infrastructure. This context is used to implement the isNeeded function. When the notifyfunction is called, a dialogue opens in the Marratech client, letting the user choose whether to accept the new media resource or not. The isAvailable function returns true for media resources which are nearby, here defined as located in the same room as the user, and which are not busy. The Q 72 Supporting Automatic Media Resource Selection Using Context-Awareness

2) Store location and Communication network status contexts service Sensor 8) Invite Marratech Simulator Manager

Context storage infrastructure 3) Context change 7) HTTP request 1) Store activity context notification

4) Get available Personal Media Marratech 5) Notify user services Client Resource Manager 6) User reply

Figure 5.3: The interaction between the components. and P functions return the quality and privacy values for each available media resource. In this prototype these values were subjectively set by the authors.

5.5.2 Interaction between components

Figure 5.3 shows the interaction between the components in the implementation when the prototype is running and a context change triggers a need for a new media resource. When the prototype is started, Marratech starts sending information about active media resource types to the context storage infrastructure (1). The sensor simulator generates a context to significantly change the media resource need in (2). When the Personal Media Resource Manager is notified about such a context change (3) it tries to find an alternative resource from the context storage infrastructure by running MRSA (4). Once a new resource has been selected the personal manager calls the notify function to ask the user to confirm the new resource (5). If the switch is approved by the user the personal manager initiates an invitation of the new resource to the session (6) by sending an HTTP request to the Marratech Manager (7) which in turn invites the resource (8).

5.5.3 Scenarios

The prototype was successfully tested on two different scenarios. The first scenario shows how an alternative communication service can be utilised if the currently used network connection can not provide enough quality. The second scenario demonstrates how the prototype can help the user find and select a more suitable communication service in order to better protect the user’s privacy. The test setup consisted of a desktop computer running a Marratech client located in the user’s office, a mobile-phone carried by the user, and an e-meeting client which was located in a conference room. For media resources the desktop computer used a microphone and speakers and as the sound was considered to have good quality, the quality index was set to Supporting Automatic Media Resource Selection Using Context-Awareness 73

6. The privacy index for this tool was set to 4 since the sound could be heard by others in the same room. The sound in the mobile-phone was considered to have lower quality, thus the quality was set to 3, but since the sound could only be heard by the user the privacy index was set to 6. Similarly to the stationary computer, the e-meeting client featured a microphone and speakers. However, since the microphone offered better quality as it featured echo supression the quality index was set to 8. On the other hand, the privacy index was considered lower than in the desktop computer case. This was because other people were present in the conference room, which was not the case with the user’s ofﬁce. Hence, the privacy index for the e- meeting client was set to 2. As previously mentioned, these indices were subjectively set by the authors.

Scenario 1: Quality degradation

In the first scenario, the user is connected to an e-meeting using the stationary computer when the audio quality suddenly drops because of bad network performance. When this context change is perceived by the Personal Media Resource Manager, it causes the personal manager to run the algorithm in order to find new media resources and the quality index for the Marratech client’s audio sink and audio source is recalculated to zero. A mobile-phone is found which satisfies the desired quality and privacy indices. The new media resources are approved by the user, the phone is invited to the session, and the user is able to communicate again.

Scenario 2: Privacy considerations

In the second scenario, the user is leaving the user’s office to temporarily visit a public conference room when another user wants to talk to the user. Consequently, as the user is away from the office, it is not possible to communicate using the Marratech client run on the desktop computer located in the office. To allow the users to talk to each other, the Personal Media Resource Manager starts looking for an alternative communication tool. An available e-meeting client and a mobile-phone are found. However, since the user has P MIN set to 4 the MRSA selects the mobile-phone even though the e-meeting client provides better quality.

5.6 Discussion

This paper considers the problem of how to automatically choose the most beneficial media resources for the user. The paper proposes to decompose the problem into manageable parts which can be solved individually. The parts which are identified include finding available media resources, creating an abstraction level for comparing the resources, and providing the necessary information needed to select appropriate media resources. Based on the information provided by these parts, the proposed algorithm, MRSA, performs media resource selection. This paper proposes that only media resources which are nearby, not busy, and which the user has permission to access are considered as available. In the implementation, resources 74 Supporting Automatic Media Resource Selection Using Context-Awareness which reside in the same room as the user are considered to be nearby and can be discovered using the context storage infrastructure. However, depending on the size of the room there may still be occasions when a user does not benefit from using two available media resources together, e.g. when a microphone is too far away from the screen that the user is currently using. Other users’ privacy preferences have not been taken into account when determining if a nearby resource is available or not. An abstraction level for comparing media resources is proposed by classifying each resource with quality, privacy, and cost indices. This solution provides a useful abstraction from both users and developers. For users it supplies a coarse-grained control interface, and for developers the issues of how to value media resources and how to choose between them are separated. What information is needed to form the indices of different media resources is not covered in this paper. In order to provide necessary information needed to select appropriate media resources, context-awareness was used. The contexts used in the proof-of-concept implementation when selecting a media resource is the user’s location, the user’s communication needs, the available media resources, and the user’s privacy and quality preferences. Since users’ location is used for access control it is important that the location can be verified. However, access control and other security issues have not been considered in this paper. In the proof-of-concept, the algorithm is used to select a new media resource every time the user changes location and when a change in network performance occurs. This works well when only a limited number of contexts can trigger the algorithm. However, in a more advanced system, which uses more contexts, it might be required to execute the algorithm in intervals instead to avoid constantly triggering it. The implementation and the two experiments performed in this paper were done to provide insight in the functionality of the algorithm. Although limited, these experiments showed that it was possible to integrate media resource selection into Marratech, the e-meeting software used in this paper. As a whole, the paper provides useful knowledge and deeper understanding of the issues involved in developing a system which automatically chooses the most beneficial media resources for the user.

5.6.1 Future work

Although a working prototype was developed there are many issues that remain to be studied. Only brief tests have been performed and the authors would like to conduct user studies to further explore how to increase the beneﬁt of automatic media resource selection. User studies could also give valuable input on how to balance autonomy with user control. In order to increase the beneﬁt of a user study, a calculation model for privacy-, quality-, and cost-indices and their prioritization should be developed. When developing this model it is also important to study which contexts should be considered when determining the privacy- , quality-, and cost-indices. It is also important to evaluate this calculation model with users in order to determine its comprehensibility. In any system which deals with personal information, it is important to ensure users’ privacy and provide appropriate security. These issues are not considered in this paper and needs Supporting Automatic Media Resource Selection Using Context-Awareness 75 to be further researched. This means ensuring that it is possible to verify a user’s location through the deployed positioning system, as well as attending to appropriate authentication and access control to the system in a ubiquitous environment. Finally, the authors would also like to develop and utilise a media gateway which supports conversion of media between a number of different formats. With better conversion support it becomes possible to utilise a wider range of media resources together in the system. The idea is to make conversion between different media resource format in an automatic fashion in order to further relieve the users.

5.6.2 Acknowledgments

This work was funded by the Centre for Distance spanning Health-care (CDH), the Centre for Distance spanning Technology (CDT), the PhD Polis project, and the C4 project which is supported by Objective 1 Norra Norrland - EU structural fund programme for Norra Norrland. 76 Part 6 homeML - An Open Standard for the Exchange of Data Within Smart Environments

Citation: Chris Nugent, Dewar Finlay, Richard Davies, Hui Wang, Huiru Zheng, Josef Hallberg, Kåre Synnes, Maurice Mulvenna, "homeML - An Open Standard for the Exchange of Data Within Smart Environ- ments",Inthe 5th International Conference On Smart Homes and Health Telematics (ICOST), pages 121–129, Nara, Japan, June 2007. Also published in Lecture Notes in Computer Science, vol. 4541, pages 121–129, 2007.

Contributions: This article was achieved together with Chris Nugent, Dewar Finlay, Richard Davies, Hui Wang, Huiru Zheng, Kåre Synnes, and Maurice Mulvenna. The primary work was undertaken by my colleagues in the University of Ulster, and Chris Nugent is the main author of this article. I joined the research at the development stage and helped refine the homeML model, although the key authors of this paper, through many interations and brainstorming, defined the homeML schema. Never- theless, I did realize the homeML schema and developed software, as well as the database, for receiving, storing, and managing data within the homeML infrastructure. Finally, I also developed the homeML browser and performed the initial tests where data was collected using movement sensors and door sensors to detect movement patterns in the office landscape, some of the results can be seen in Figure 2.2.

77 homeML - An Open Standard for the Exchange of Data Within Smart Environments 79

homeML - An Open Standard for the Exchange of Data Within Smart Environments

Chris Nugent1, Dewar Finlay1, Richard Davies1, Hui Wang1, Huiru Zheng1, Josef Hallberg2, Kåre Synnes2, Maurice Mulvenna1 1 School of Computing and Mathematics, University of Ulster, N. Ireland {cd.nugent, d.ﬁnlay, rj.davies, hy.wang, h.zheng, md.mulvenna}@ulster.ac.uk 2 Luleå University of Technology, Sweden {josef.hallberg, kare.synnes}@ltu.se

Abstract

This work describes a potential solution to the problems caused by the heterogeneous nature of the data which may be collected within smart home environments. Such information may be generated at an intra- or inter-institutional level following laboratory testing or based on in- situ evaluations. We offer a solution to this problem in the form of a system/application/format independent means of storing such data. This approach will inevitably support the exchange of data within the research community and form the basis of the establishment of an openly accessible data repository. Within this abstract we present the outline design of homeML, an XML based schema for representation of information within smart homes and through exemplars demonstrate the potential of such an approach. An example of the typical type of software browser required for the data representation is also presented.

6.1 Introduction

As the population continues to grow and the percentage of the elderly within the population also increases, there is a growing demand to develop and deploy technical solutions within the home environment to support levels of independence. This has the inevitable and indeed desired effect of extending the period of time a person can remain in their own living environment prior to the potential need of institutionalisation. It can also offer a positive impact on measures associated with quality of life. Solutions and services which may be deployed can take the form of, for example, remote vital signs monitoring [35], activity tracking [112] and assistive technology to control ambient settings [54]. In addition, devices and sensors may have the ability to monitor and record a person’s movement within a room, to asses pressure values to gauge if a person is sitting in a specific chair or even in bed, or have the ability to measure the flow of water within the bathroom and kitchen. These types of sensory information are complemented by information from a series of specific supporting devices and as such may be considered as the underlying technology, upon which services are deployed by healthcare professionals. 80 homeML - An Open Standard for the Exchange of Data Within Smart Environments

Given the importance of this domain and the huge beneﬁts to be gained it has attracted much attention in the past decade from both commercial, healthcare and research perspectives. The net result has been the largely independent development of a series of specialised services by differing organisations, resulting in the generation of a multitude of data, which in effect is largely heterogeneous. Analysis of this data, depicting how users interact with the technology and the environment, provides useful information relating to lifestyle trends and how the environment may be adapted to improve the user’s experience [115]. Taking the issues associated with the heterogeneous nature of the storage and the distribution of the data into consideration there is a growing need and indeed opportunity within the general smart environment research community to promote the development of a cross- system standard to support information exchange and improve accessibility to and analysis of the collected data. Adoption of such an approach could be deployed at both intra- and inter-institutional levels. Historically, the storage and distribution of information related to a person’s activities within smart environments has been based on different formats and generally developed on a project-by-project basis. The research presented in this paper seeks to develop an XML-based system/application/format independent model, referred to as homeML, for representing and exchanging diverse data recorded within a range of service scenarios, monitoring devices and in-situ evaluations within the home environment.

6.2 Background

Although research efforts within the domain of smart homes have been prolific in the past decade it is becoming increasingly evident that there does not exist any centrally and freely available data repository of recorded information or open standards/protocols to follow in the acquisition and storage of data in such environments. This may hinder the exchange of data at both intra- and inter- organisational levels and its ensuing analysis. Efforts have been made to address this issue for complex physiologic signals through the commonly used PhysioNet resource [114]. Another example is the OpenECG portal [101] which aims to promote the consistent use of format and communication standards for electrocardiograms. Similar resources of this nature would be greatly beneficial within the domain of smart environments. The issue and need to develop an open standard for the exchange of data should not be confused with the already existing standards and protocols which facilitate the seamless integration of numerous devices and services within the one environment. These efforts have strived to address the problems associated with compatibility issues between differing types of devices. Two of the most prominent standards which have emerged within this domain are the Open standards gateway initiative (OSGi) [103] and Konnex [71]. The motivation behind these standards has been to make all aspects of the deployment of home based technology simpler. This includes the connection of systems, the transfer of data and the management and provisioning of network devices. It is upon these services that the requirements to specify the structure of the data to be stored, as proposed by homeML, rests. Efforts have also been focused towards the development of standards to support the data exchange of patient information in the form of electronic patient records (EPRs) on both intra- and inter-institutional levels. The most notably work in this domain can be found homeML - An Open Standard for the Exchange of Data Within Smart Environments 81 associated with the efforts of HL7 [53]. The mission of HL7 is to create standards for the exchange, management and integration of electronic healthcare information. Since 1997 HL7 has been recognised as the standard for electronic exchange of historical and administrative data in health services world wide. Although having widespread acceptability such standards do not facilitate the establishment of a data structure for data within smart environments, its recording, exchange or storage. One of the key technology recommendations within HL7 is the use of the eXtensible Markup Language (XML) [40]. XML has gained much attention in recent years and of particular relevance has been described as an acceptable means to represent biomedical data. XML has also become a widely established solution for the expressing of a syntax for data and the exchange of data via the Internet. The main benefits of employing XML relate to its platform, vendor and application independence. It is also considered to offer an easy to follow hierarchical data structure. Within the general area of home based healthcare monitoring a limited number of studies have reported application of this technology. Wang et al. [145] have demonstrated how problems with the exchange of electrocardiogram data can be addressed via an open standard development based on XML. Feki et al. [42] have presented models for the development of XML based objects to support interaction between modules in environments offering support for those with disabilities. Finally, Ohmori et al. [100] have presented an XML based multimedia data acquisition and inquiry system for wearable computers. Although these studies have been successful in their own application domains, the issues of addressing the needs of a truly open standard for the exchange of data within smart environments has not been addressed. Through the use of such a standard it will become possible that all those involved within the research domain will have the opportunity to record and store their data in a common format which will support the establishment of a web based repository which can be freely accessible. In addition, flexibility in such a standard will allow it to evolve as the needs of users and their further understanding of smart environments develops. It is the applicability of XML to address the problems of data representation and storage within smart environments that we explore further in the following Sections.

6.3 Methods

As previously stated, there is a growing need to address the heterogeneity of the data generated within smart environments in addition to including necessary information such as patient details, any clinical interventions and details of any vital signs which may also have been recorded. In the first instance we have designed homeML as a first attempt to offer a solution for an open standard for the storage and exchange of data within smart environments. Our first step in this process has been to design homeML as a series of hierarchical data trees (Figure 6.3). We have supplemented this graphical representation with Figures 6.1 and 6.2 which describe the elements and attributes within the aforementioned models. Each smart home starts with a root element SmartHome, which is uniquely identified by its attribute homeID. The main components for each smart home are DateRecordCreated, SmartHomeInfo, Room, InhabitantDetails and Comment. Each smart home can only have one set of each component aside from Room elements, of which a multiple number are allowed. It 82 homeML - An Open Standard for the Exchange of Data Within Smart Environments

SmartHome The root element for XML based smart home data Element/ Description Required/ Data type Example Attribute Optional homeID A unique Required Integer 23165146 identifier for the SmartHome DateRecordCreated The creation date/ Required Date 2007/01/ time of the record 10T12:30:00 SmartHomeInfo General Required Requires information about element the Smart home description InhabitantDetails Personal Required Requires information on element inhabitants description Room Definition of a Required Requires room within the element smart home description Comment Additional Optional String Monitors comments about glucose level the smart home and temp.

Figure 6.1: Description of SmartHome element based on the hierarchical representation of homeML in Figure 6.3 should be noted that, in line with practical deployment of smart homes, we have limited this schema to single occupancy only. It is expected that issues pertaining to multiple occupancy will be addressed in following versions. In the following paragraphs we provide a brief description of each of the elements included in the tree structure. DateRecordCreated contains the date the XML file in question was created. SmartHome- Info contains information on the location, region and details of those responsible for the smart home. This is contained within the elements Address, Jurisdiction (regional funding author- ity), CareProvider (care agency responsible for the home) and ContactPerson (individual immediately responsible). This is information common to the smart home which does not differ between rooms and is important especially in the instances of exchanging data between organisations. Room as the name suggests represents a room in the smart home. There can be a number of rooms within the smart home and these are defined using the elements as shown in Figure 6.3. These elements include details of when the data was recorded (DateOfRecording), a RoomDescription (e.g. bedroom, bathroom, kitchen, etc), the Floor, which specifies the floor (level) which the room is on. There are two further elements which deserve further attention. The first of these is RoomParam which describes the basic make up of the room. This is represented using five further elements which specify the number of doors and windows and the size of the room (based on X, Y and Z values), initially we assume a room is square. The final element within Room is the Device, of which there can be any number. This element will hold information relating to devices that are fixed in each room. Typical examples of such devices would be motion detectors, water flow sensors, temperature sensors and door homeML - An Open Standard for the Exchange of Data Within Smart Environments 83

Device The Device element for XML based smart home data Element/Attribute Description Required/ Data type Example Optional deviceID A unique device Required Integer 13165146 identifier Description General Optional String The device description of the measures the device room temp. DeviceLocation Details the Optional XPos absolute float 4.5 YPos coordinates of the float 1.9 ZPos device float 6.2 DeviceType Details the type Required String Temp of device Sensor Units Measurement unit Required String Celsius RealTimeInfo Specifies sample Optional runID rate, as well as int 25341 SampleRate start and end time, float 10.1 StartTime identified by the Date EndTime runID attribute Date Event Describes an Optional eventID event from the int 64721 TimeStamp device Date Data String 175

Figure 6.2: Description of Device element based on the hierarchical representation of homeML in Figure 6.3 sensors. The Device is made up of several elements which are described in Figure 6.2. InhabitantDetails provides personal information relating to the person who is living within the home environment and their Care Plan. Depending on the ethical regulations at an institutional level, varying amounts of this information may be considered as optional. In addition, from a data exchange perspective, this information will not be made available and will be managed according to each institution’s ethical guidelines. InhabitantDetails also contains AmbulatoryDevices which provides details on wearable health systems (e.g. Holter monitors and vital signs equipment) that the person may require to monitor their health status. The make up of AmbulatoryDevices (its sub-elements) is similar to that of Device which appears as an element of Room. The subtle differences are the optional elements QuantizationRes- olution (e.g. 1 volt), and ElectrodePlacement (general area or body part where electrode is placed) which replaces the location. Finally, Comment as the name suggests can contain non-speciﬁed information about the ﬁle.

6.4 Evaluation of the Model

Based on the design of homeML we have now produced the ﬁrst version of the XML based schema. With this it is now possible to deploy a common standard at both intra- and inter- 84 homeML - An Open Standard for the Exchange of Data Within Smart Environments institutional levels for the recording of information collected within smart environments (either laboratory based experiments or real living environments) which will facilitate the open exchange of data and will assist in moving one step closer towards the development of a common data repository for use by the entire research community. To facilitate uptake of this approach we will make available, through a series of web based resources, the XML schema along with documentation providing guidance on use and steps to follow to make recommendations for changes to the current version (http://trail.ulster.ac.uk/HomeML/). In addition to providing the schema for homeML we are also planning to make available a suite of open source tools to assist users in exploiting the general concepts. In the ﬁrst instance this will take the form of a viewer or browser which will provide a means to display the data recorded. Figure 6.4 provides an example of our vision for the development of such a tool. Figure 6.4 (a) provides an exemplar of an excerpt of data according to the homeML schema. Figure 6.4 (b) provides a prototype representation of how we envisage the homeML browser to be represented. On the left hand side we can view the tree structure (which can be mapped to our design in Figure 6.3). This can be expanded and collapsed at any level. The interface on the right hand side will have the ability to represent the layout of the room, the devices which have been included in the room and also to plot any time series data which may have been recorded. In this example, the browser is providing a means to plot temperature values within the room.

6.5 Conclusion

The ability to embed sensors within the home environment coupled with the increasing prevalence of wearable health care solutions for vital signs monitoring offers huge potential to unobtrusively monitor the activities of a person within their home environment and subsequently adapt the environment to offer a personalised living space. The types of devices/sensors/services which may be present within the home environment and the data they generate vary widely. It has been the aim of the current study to address the problems caused by the heterogeneous nature of the data generated within smart homes through the development of an open standard for data storage and exchange. These developments have been realised through the presentation of homeML, an XML based schema. This approach provides a common platform for the exchange of data in addition to offering suggestions for an environment for representation of the data through the proposed open source homeML browser. Developments are currently underway to develop the first version of this open source browser. Access to the homeML source files will be made available through web based resources (http://trail.ulster.ac.uk/HomeML/) or can be obtained directly by contacting the authors of the paper. With this we hope to inform the community of these developments and maintain the user-driven notion of using XML. Finally, it is our intention, in the first instance, to secure collaborations with a small number of organisations who would be willing to evaluate the concepts of homeML and begin to establish a repository which would become widely available within the general research community. This will allow us to ascertain if homeML in its current form contains all the necessary information required to support meaningful analysis of persons within smart home environments. homeML - An Open Standard for the Exchange of Data Within Smart Environments 85

SmartHome homeID Key + Must exist at least once DateRecordCreated * Can exist any number of times ? Optional SmartHomeInfo Nodes without any sign must exist once + Room roomID

DateOfRecording

RoomDescription ? RoomParam

NumWindows NumDoors XDimension YDimension ZDimension

Floor * Device deviceID ? Description ? DeviceLocation

XPos YPos ZPos

DeviceType

Units

* RealTimeInformation runID

SampleRate StartTime EndTime

+ Event eventID

TimeStamp Data

InhabitantDetails patientID ? Name * AmbulatoryDevices deviceID ? Description

DeviceType

Units ? QuantizationResolution ? ElectrodePlacement

* RealTimeInformation runID

SampleRate StartTime EndTime

+ Event eventID

TimeStamp Data ? Comment ? CarePlan ? Comment

Figure 6.3: Hierarchical representation of homeML as a series of Tree diagrams. 86 homeML - An Open Standard for the Exchange of Data Within Smart Environments

2007-01-04T12:30:00.05

16J27

Community and Hospitals Trust Medical science Dr. Martin 2007-01-04T16:35:45.27 bed room 2 1 12 10 2.5 …………………………. …………………………. ………………………….

Figure (a)

Window 1

Temperature Sensor

Window 2 Window (13165146) 10 feet

Door 12 feet

26 28 28 28 26 26 26 27 22

Temperature 20

Time

Figure (b)

Figure 6.4: Hierarchical representation of homeML as a series of Tree diagrams. Part 7

HomeRuleML - A Model for the Exchange of Decision Support Rules Within Smart Environments

Citation: Josef Hallberg, Chris Nugent, Richard Davies, Kåre Synnes, Mark Donnelly, Dewar Finlay, Maurice Mulvenna, "HomeRuleML - A Model for the Exchange of Decision Support Rules Within Smart Environments",Inthe 3rd Annual IEEE Conference on Automation Science and Engineering (CASE), pages 513–520, Scottsdale, Arizona, USA, September 2007.

Contributions: I am the main author of this article, I initiated the work and I also developed the model itself. Nevertheless, HomeRuleML is based on the research undertaken in homeML, and the article was achieved together with Chris Nugent, Richard Davies, Kåre Synnes, Mark Donnelly, De- war Finlay, and Maurice Mulvenna.

HomeRuleML - A Model for the Exchange of Decision Support Rules... 89

HomeRuleML - A Model for the Exchange of Decision Support Rules Within Smart Environments

Josef Hallberg1, Chris Nugent2, Richard Davies2, Kåre Synnes1, Mark Donnelly2, Dewar Finlay2, Maurice Mulvenna2 1 Luleå University of Technology, Sweden {josef.hallberg, kare.synnes}@ltu.se 2 School of Computing and Mathematics, University of Ulster, N. Ireland {cd.nugent, rj.davies, mp.donnelly, d.ﬁnlay, md.mulvenna}@ulster.ac.uk

Abstract

The demands for smart environments, which can help to facilitate as well as monitor independent living are increasing. With this comes a desire for decision support rules to process data recorded from such environments. However, testing and evaluating rules can be both time- consuming and indeed stressful for the inhabitants. Within this paper we propose a model, referred to as HomeRuleML, for representing decision support rules for smart environments. The motivating factor behind such a proposal is to provide a widely and freely accessible set of rules which can be openly used and exchanged within the research domain and beyond. This model has the potential to decrease the time required for deployment, and inevitability improve the inhabitants’ quality of life. In the paper we explain in detail the structure of adopting this approach and also provide an indication of the typical types of software tools required for its use.

7.1 Introduction

"There is no place like home" is a well known phrase which becomes more pertinent in this day of age as many people choose and indeed prefer to live at home as they become older [28], or suffer from an illness. Why should people not be able to stay at home if they want to and if they are in good health? Facilitating the need of these people to stay at home has created a greater need for smart living environments which can help assist in daily activities and support increased levels of independence. These smart environments have the inevitable and indeed desired effect of extending the period of time a person can remain in their own living environment prior to the potential need of institutionalisation. Customizing an environment to meet the specific needs of an individual can be both time consuming and stressful. Even if personalities between individuals are unique, their impairments or illnesses are often similar. Therefore, in the current work we hypothesise that using standards and models, both for storing data from the environment and for creating rules, can help reduce the time required to customise an environment in addition to the stress imposed 90 HomeRuleML - A Model for the Exchange of Decision Support Rules... on the inhabitant. The net effect of such an approach would provide the desired effect of making these smart environments easier to deploy, more reliable, and reduce costs. Given the importance of this domain and the huge benefits to be gained it has attracted much attention in the past decade from both commercial, healthcare and research perspectives. The result has been the development of a number of expert systems and rule-based systems for the analysis of information recorded within smart environments [137, 70, 34]. Nevertheless, few attempts have been made to achieve a cross-system standard for decision support rules intended for smart environments where environmental and sensor-based information should be taken into account. This work looks at creating a standard for the storage and exchange of decision support rules to be used in smart environments. The model, which is referred to as HomeRuleML, builds on previous work by the authors [96] which offers a standard (HomeML) which can be used in the storage and exchange of information recorded within such environments. Designing a rule must be a simple process since the technical developers are not always the people with the most knowledge about the needs of the inhabitant. The rules must also have sufficient details for the technical developers to configure them to the available resources (e.g. sensors) in the environment. In addition, it must be possible to integrate the rules dynamically into a smart environment. These requirements lead to the following research questions which the authors seek to answer: • How can rules be generated and shared in a simple manner? • How can a narrative text describing a rule be transformed into a HomeRuleML rule? • How can HomeRuleML rules be parsed and understood by a smart environment application? The rest of the paper is organized as follows: Section 7.2 gives a brief background to the work presented in this article, in addition to some related work. Section 7.3 presents the method used in developing HomeRuleML, as well as the model itself. Section 7.4 explains the evaluation strategy and presents an example scenario. Section 7.5 discusses the research questions in more detail, and Section 7.6 concludes and summarizes the results and gives an overview of the future for HomeRuleML.

7.2 Background

Trying to develop a common standard for representing decision support rules is not a new concept. For example, within the ﬁnancial world this has been a proliﬁc area of interest for many years where decision support rules are used for recommendations on stock market trends [74, 45]. The driving force behind a standard for rules for use within this scenario is because parameters (e.g. stock prices etc.) are constantly changing and sometimes the rules are required to be altered in order to take these changes into account. The same driving force exists for rules in smart environments where it is not desirable to update the entire application to add a rule. In addition being able to create a repository of rules for exchange between researchers is also an important factor. HomeRuleML - A Model for the Exchange of Decision Support Rules... 91

In smart environments the attempts to develop a standard for decision support rules have been sparse. There are, however, a few examples. These include the first order logic used in Gaia [118], FACTS [140], and CADEL [92]. Although these examples address the dynamic nature of rules they are not developed for the purpose of sharing between smart environments. The rules are also very high level and lack detailed information about sensors in the environment and the environment itself. The general sensor abstractions provided can be useful for distribution, but will then require an additional layer which binds the sensor abstractions to specific sensors. This is different from the approach taken with HomeRuleML, which provides the necessary information for binding sensor data to a specificsensor. The HomeRuleML model in this article is based on HomeML [96], a standard for representing, exchanging and storing diverse data recorded within smart environments. HomeML provides detailed information about sensors (sensor events, sensor location, etc.), the environment (room information, room purpose, etc.) and the inhabitant (including any wearable sensors). HomeML can therefore be compared to a context-storage platform built upon XML with the purpose of sharing data between smart environments. Given the proposal to base HomeRuleML upon the concepts of HomeML the information in the HomeRuleML model can be kept simple and easy to understand, and it will refer to the additional information stored in HomeML.

SmartHome homeID + Must exist at least once + * Can exist any number of times Room roomID ? Optional * Nodes without any sign must exist once Device deviceID ? Description ? DeviceLocation

XPos YPos ZPos

DeviceType

Units

* RealTimeInformation runID

SampleRate StartTime EndTime

+ Event eventID

TimeStamp Data

Figure 7.1: Excerpt of the HomeML deﬁnition.

Figure 7.1 shows the most relevant nodes of the HomeML definition. Apart from general information about the smart-environment and the rooms, it has a Device subtree which stores information about various sensors in the environment. Each device is identified by a deviceID, and contains a general description, information about location, device type (e.g. MovementSensor, TemperatureSensor, etc.), and which unit is being used for the sensor-data (e.g. Celcius, State, etc.). Furthermore, there is information for separating between different trials or "runs" (RealTimeInformation) containing sample rate of the sensor, as well as start 92 HomeRuleML - A Model for the Exchange of Decision Support Rules... and stop time. Finally there is room to store sensor data in the Event node. Each event is identified by an eventID but also contains a time stamp. HomeRuleML as well as HomeML are created using the eXtensible Markup Language (XML) [40]. XML has gained much attention in recent years and has become a widely established solution for the expressing of syntax for data and the exchange of data via the Internet. The main benefits of employing XML relate to its platform, vendor and application independence. It is also considered to offer an easy to follow hierarchical data structure. There are also XML standards such as RuleML [123], and Drools [67] which implement the recent Java Rule Engine standard [68]. Nevertheless, these XML standards also lack support in terms of the specific details needed for a rule set for use in a smart environment. Although there have been several successful rule-based systems, none of these have addressed the issues and needs of a truly open standard for the exchange of decision support rules intended for smart environments. Such a standard would facilitate the open exchange of data and will assist in moving one step closer towards the development of a common rule repository for use by the entire research community. It will also help to decrease the complexity of deploying smart environments. XML provides a flexible base for storing and modifying rules. It also offers a suitable standard for information exchange over the web.

7.3 Methods

As previously stated, it is important to facilitate the exchange of rules between both researchers and smart environments and to simplify the deployment process for smart environments. Providing an openly accessible repository with tested and verified rules is one means that can be offered to help achieve this goal. HomeRuleML has been designed as a first attempt to offer a solution for an open standard for decision support rules within smart environments. For sufficient flexibility and usability the model should support the application of logical expressions to sensor data, the possibility of defining the order of events, and to define time constraints. Furthermore, it should incorporate functionality for the reuse of more generic rules to simplify the creation of more complex rules. With HomeRuleML it was decided that abstractions may be made in sensor data and contexts at a lower level, but references to sensors, context providers, and other rules, will be made explicit through the use of identification numbers. The first step in achieving the aforementioned requirements has been to design Home- RuleML as a series of hierarchical data trees (Figures 7.4 and 7.5). These graphical representations have been supplemented with Figures 7.2 and 7.3 which describe the elements and attributes within the aforementioned models.

7.3.1 The Rule Tree

Each rule starts with a root element Rule (Figure 7.4 and 7.2), which is uniquely identiﬁed by its attribute ruleID. The main components for each rule are Description, DateRecordCreated, HomeRuleML - A Model for the Exchange of Decision Support Rules... 93

Required/ Element/Attribute Description DataType Example Optional Rule The root element for XML based home rule ruleID A unique identifier for the rule (attribute) Required Integer 1001 Description General description of the rule Required String Medication Management – This rule sends an alarm if the medication isn’t taken at the correct times DateRecordCreated The creation date/time of the record Required TimeStamp 2007-04-23 11:56:05 Operation A reference to an Operation component Required Reference See Operation table (Table II) Outcome Specifies the rule’s outcome Required Value The value of the rule if the result is true Optional String Food Taken Action The action to be taken if the result is true Optional Device The device involved in the action Required deviceID The ID of the device involved (attribute) Required Integer 2001 SetValue The resulting value the device should take Required String Off, Alarm, etc.

Figure 7.2: Description of Rule element based on the hierarchical representation of Home- RuleML in Figure 7.4

Required/ Element/Attribute Description DataType Example Optional Operation The root element for Operations operator The logical operation for the element Required String AND, OR, NOT, NAND, NOR, XOR, (attribute) ORDER Time Specifies time conditions for the Operation Optional condition The logical operator to be applied (attribute) Required String GT, LT, EQ, LEQ, GEQ, NOT Value The time value to compare with (in seconds) Required Integer 3600 (seconds) Device A device involved in the Operation Optional deviceID A unique device identifier (attribute) Required Integer 2002 Description General description of the device Optional String Bed Occupancy Sensor DeviceType Details the type of the device Required String PressureSensor Units Measurement unit Required String State, Celsius, etc. Condition The condition for the events of the Device Required condition The logical operator to be applied (attribute) Required String GT, LT, EQ, LEQ, GEQ, NOT Value The value to compare with Required Varies Open, 20, etc. Ref: Operation A reference to the root element for Operation Optional RuleReference A reference to a different HomeRuleML rule Optional ruleID The unique identifier for the rule (attribute) Required Integer 1002 Description General description of the rule Optional String Food Sensor Condition The condition for the Rule Required condition The logical operator to be applied (attribute) Required String GT, LT, EQ, LEQ, GEQ, NOT Value The value to compare with Required String Food Taken

Figure 7.3: Description of Operation element based on the hierarchical representation of HomeRuleML in Figure 7.5

Operation, and Outcome. Each rule can only have one set of each component (Figure 7.2). Description contains the description and purpose of the rule, and also what kind of result or action can be expected as an outcome. DateRecordCreated contains the date that the XML ﬁle in question was created. The Operation component contains logical operators which can be applied to devices, operations, or rules. It should be noted that Operation is a reference, in order to accommodate recursive behaviour of operations, and will be discussed later (see Figure 7.3). The Outcome is, as its name suggests, the outcome of the rule itself and is of interest if the Boolean result from the Operation is true.AnOutcome can consist of a Value and an Action.TheValue is a String value and is used primarily when the rule is being used as part of another rule. It also helps explain the purpose of the Rule.TheAction can consist of a number of Device components identiﬁed by their deviceID attribute. Each Device contains a SetValue element which is the value to be applied to the device in question as a result of the 94 HomeRuleML - A Model for the Exchange of Decision Support Rules... rule and the Operation being true.

Rule ruleID + Must exist at least once * Can exist any number of times ? Optional Ref: Reference to another node Description Nodes without any sign must exist once DateRecordCreated

Ref: Operation

Outcome

? Value

? Action + Device deviceID

SetValue

Figure 7.4: Hierarchical representation of HomeRuleML Rule tree

7.3.2 The Operation Tree

The Operation is a support component for rules (Figure 7.5 and 7.3). It is a recursive schema which makes it possible to facilitate nested operations and rules in order to form complex expressions. The root element is Operation which has an operator attribute that can have the following logical operators: AND, OR, NOT, NAND (neither), NOR, XOR (either one or the other, but not both) and a special operator ORDER which makes it possible to define a sequence of events. The operator will be applied to the underlying elements, but will not traverse the tree to apply itself on nested elements. The main components for each operation are Time, Device, Operation,andRule (see Fig- ure 7.3). The Time component is optional and is for defining temporal logic to be applied to the elements of the current Operation. The attribute of the Time component is a condition which can have the following operators: GT (greater than), LT (less than), EQ (equals), GEQ (greater than or equal to), LEQ (less than or equal to), and NOT. The condition will be applied to the Value subcomponent which is defined as time in seconds and the result will be a Boolean value. The Device component, identified by its deviceID attribute, contains an optional Descrip- tion (describes the purpose and placement of the device), a DeviceType (e.g Temperature- Sensor), Units (e.g. Celcius), and one or more Condition components. The Condition has a condition attribute which has the same operators as the attribute for the Time component. The condition will be applied to the event values which can be found (using the deviceID attribute) in the HomeML standard and compared to the Value subcomponent, the result will be a Boolean value. If there are more than one Condition components an OR logic operation will be applied to the results from each Condition in order to form a single Boolean result. HomeRuleML - A Model for the Exchange of Decision Support Rules... 95

Operation operator + Must exist at least once * Can exist any number of times ? Optional Ref: Reference to another node ? Time condition Nodes without any sign must exist once

Value

* Device deviceID

? Description

DeviceType

Units

+ Condition condition

Value

* Ref: Operation

* RuleReference ruleID

? Description

Condition condition

Value

Figure 7.5: Hierarchical representation of HomeRuleML Operation tree

Lastly there is a reference component, Operation, which makes it possible to create nested expressions. Similarly to a reference, there is a RuleReference component which contains the ruleID attribute of another Rule (Figure 7.2) to use as part of the Operation. Each Rule which is to be used in a nested Operation is required to have its Value in the Outcome set. The Description and the Condition components in the RuleReference are similar to previous occurrences and the condition operator will be applied as a String operator. In this manner the operator can be applied to rules and operations in a similar way. The result from all Operation components will always be a Boolean value.

7.4 Evaluation of the model

Based on the design of HomeRuleML the ﬁrst version of the XML based schema has been produced. To facilitate uptake of this model, the XML schema along with documentation providing guidance on use and steps to follow to make recommendations for changes to the current version, will be made available through a series of web based resources (http://trail.ulster.ac.uk/HomeML/ ). 96 HomeRuleML - A Model for the Exchange of Decision Support Rules...

In addition to providing the schema for HomeRuleML the authors are also planning to make available a suite of open source tools to assist users in exploiting the general concept. In the first instance this includes the HomeML standard (XML schema) for storing and sharing data from smart environments. There are also a number of assistive tools such as a browser and data editor which will help display and edit/populate HomeML data, a SQL database for efficient data storage and access on site, and tools for generating the HomeML files from the database. In addition a graphical interface for creating rules [93], will also be made available.

7.4.1 Example scenario

The creation of rules can be performed in many different ways. One way would be to use a graphical tool to generate the rule and then parse the graphical representation and generate the desired XML code. However, having to develop a rule from a narrative is a common situation. Of course a graphical tool could be used in this situation as well [93], but creating the desired XML code directly from the narrative might be preferred by some. HomeRuleML is designed to be self explanatory to anyone with basic knowledge of XML, and hence the creation of rules can be performed directly in XML. Below is a short example of a narrative of a typical situation to be managed within a smart environment. This serves to offer a useful exercise in demonstrating the creation of the XML code: Medication Management - A person must take their medication within 1 hour after getting out of bed. Within this period they must also ensure that they have had something to eat prior to taking the medication. The first step to develop a rule would be to identify all the information which must be recorded. Detecting bed activity can be performed with a bed sensor and detecting if someone has taken their medication can be performed by using a smart pill dispenser [95]. However, detecting if someone has eaten might require a number of different sensor recordings. Here we assume that it is sufficient to detect if there has been movement in the kitchen and the fridge door has been opened in order to determine that someone has eaten. Hence, the XML representation to ascertain if food has been taken could be expressed as shown in Figure 7.6(a) (note that we choose to use an outcome value so this rule can be used within the main rule). The narrative describes a certain order in which the events must happen, as well as some time limitation for these events. First the person needs to get out of bed, and from this event the person needs to first eat, and then take their medication, all within one hour (or 3600 seconds). If these steps are not fulfilled in the specified order or within the time constraints the rule should trigger an action. In this case we trigger an alarm on one of the devices in the environment. The XML representation for this part of the scenario, using the Food Sensor rule as previously presented (Figure 7.6(a)), can be represented as shown in Figure 7.6(b). Representation of the medication management rule in the HomeRuleML format means that it could be stored on a central repository and openly shared with others. The goal is that, assuming the same device IDs are used for sensors, deployment of such a rule would simply work without any need for change in their application. In cases where device IDs HomeRuleML - A Model for the Exchange of Decision Support Rules... 97

xsi:noNamespaceSchemaLocation="file:HRML.xsd"> Medication Management Food Sensor 2007-04-23 11:56:05 2007-04-23 11:58:05 3600 Bed Occupancy Sensor PressureSensor State Kitchen Movement Sensor Out of Bed Movement Sensor Food Sensor State Food Taken Movement Pill Dispenser TiltSensor State Fridgedoor Sensor Pill Taken DoorSensor State Open Alarm – Medication Alert Food Taken

(a) The HomeRuleML representation of the (b) The HomeRuleML representation of the Medica- Food Sensor rule tion Management rule

Figure 7.6: HomeRuleML representations of sample rules mismatch the descriptions provided in HomeRuleML should be clear enough that it is clear which device ID to include. Note that a rule can also consist of other more generic rules (e.g. Food Sensor) which is good for hiding the complexity of "lesser" activities.

7.5 Discussion

Creating rules is often associated with a complex procedure. In many cases this results in rules being created by developers and not actually by the people with the insight and domain knowledge [93]. The desire is of course to make the rule construction so simple that anyone could do it, yet so descriptive that the end result could be used as a component in an existing model monitoring a smart environment. In parallel to the HomeRuleML model a graphical front-end [93] has been developed by the authors. Initial evaluations have shown that this interface is a viable and appropriate method for healthcare professionals to create rules for smart environments. However, turning this graphical representation into a HomeRuleML representation is a different matter. Cur- rently the rules are stored graphically and a parser needs to be developed in order to generate 98 HomeRuleML - A Model for the Exchange of Decision Support Rules...

Figure 7.7: An example of the HomeRuleML browser

HomeRuleML code from it. Given that the schema is well defined and HomeCI is created for making rules for smart environments it should be possible. Another way to make the creation of rules easier would be to create a browser which is based on the HomeRuleML schema. By using such a tool it would then be possible to edit the rules in this browser, which would make it possible for people without a programming background to design and update the rule (Figure 7.7). In addition, there are already a number of powerful XML authoring tools on the market which would simplify the generation of HomeRuleML code. Once a HomeRuleML file is created and validated it is desirable to be able to plug the rule into a system without having to modify the application. As previously mentioned use of the Java Rule Engine [68] will help to make this possible. HomeRuleML is also built on standard logic operators and the hierarchic structure of XML; hence it makes it easy to parse the elements in a logical order. The Simple API for XML [133] is an event driven parser which makes it possible to get events in both the beginning and the end of a node. This would mean that HomeRuleML code would yield a logical expression with the operator first, followed by the parameters (the sub-elements). Written down it would look something like AND (A, B, C), which means that the AND operator will be applied to elements A, B and C. Taking advantage of the hierarchical structure of XML and the Java Rule Engine it is possible to parse HomeRuleML files, and make them understood by a smart environment application.

7.6 Conclusions and future work

Being able to share decision support rules for smart environments offers great potential to decrease deployment time, increase reliability, and decrease cost. Even if individuals are different, they often share similar impairments or illnesses, which would suggest that they would require similar support from the environment. Such support can greatly improve the quality of life of this cohort of people. To facilitate this will require a number of thoroughly tested and evaluated rules. HomeRuleML - A Model for the Exchange of Decision Support Rules... 99

This article has presented HomeRuleML, a model for creating decision support rules for use within smart environments. Usage of HomeRuleML offers the potential to create a repository of rules which can be easily deployed. HomeRuleML has been realized by using the strengths of XML, such as the hierarchical structure and suitability for web sharing, and as such has resulted in a XML based schema. Access to the HomeRuleML source files will be made available through web based resources or can be obtained directly by contacting the authors of this paper. With this we hope to inform the community of these developments and maintain the user-driven notion of using XML. Furthermore, we plan to further develop and evaluate HomeRuleML, as well as create the necessary implementation for making it possible to dynamically add HomeRuleML rules to an existing smart environment application. Finally, it is our intention to secure a number of collaborators who can help evaluate the model as well as help create the first set of rules in order to establish a repository which would become widely available within the general research community. It is our belief that this will help further improve HomeRuleML for the benefit of both research community and the inhabitants of smart environments.

7.7 Acknowledgements

This work has been partly supported through the Nestling Technologies Project, the MyMates Project, and the Centre for Distance-Spanning Healthcare (CDH). 100 Part 8

HomeCI - A visual editor for healthcare professionals in the design of home based care

Citation: Chris Nugent, Richard Davies, Josef Hallberg, Mark Donnelly, Kåre Synnes, Michael Poland, Jonathan Wallace, Dewar Finlay, Maurice Mulvenna, David Craig, "HomeCI - A visual editor for healthcare professionals in the design of home based care",Inthe 29th An- nual IEEE Conference on Engineering in Medicine and Biology Society (EMBC), pages 2787–2790, Lyon, France, August 2007.

Contributions: This article was achieved together with Chris Nugent, Richard Davies, Mark Donnelly, Kåre Synnes, Michael Poland, Jonathan Wallace, De- war Finlay, Maurice Mulvenna, and David Craig. The main author of this article is Chris Nugent who also initiated the research, however, the visual editor was developed by Michael Poland. My main contribution was with the evaluation and testing of the HomeCI tool.

101

HomeCI - A visual editor for healthcare professionals in the design of home based care 103

HomeCI - A visual editor for healthcare professionals in the design of home based care

Chris Nugent1, Richard Davies1, Josef Hallberg2, Mark Donnelly1, Kåre Synnes2, Michael Poland1, Jonathan Wallace1, Dewar Finlay1, Maurice Mulvenna1, David Craig3 1 School of Computing and Mathematics, University of Ulster, N. Ireland 2 Luleå University of Technology, Sweden 3 Belfast City Hospital / Queen’s University of Belfast, N. Ireland

Abstract

The demands of introducing a more practical means of managing and monitoring technology within the home environment to support independent living are increasing. Within this paper we present a prototype solution, referred to as HomeCI, which allows healthcare professionals to establish the conditions/rules within which technology in the home should operate. The HomeCI concept is based on the use of visual notation and has been designed for use by healthcare professionals with a non technical background. Within the paper we present the design of the ﬁrst version of the HomeCI visual editor and present the results of a usability study conducted on 4 healthcare professionals.

8.1 Introduction

Advances in healthcare technologies which provide the ability of home based vital sign monitoring in conjunction with wearable healthcare systems and smart environments are beginning to revolutionise the way healthcare is being delivered and managed. This comes at a time when the general population have a genuine interest to take more control of their own healthcare management and indeed wish to have a deeper insight into its long term impact. This, coupled with demographic changes in the population and increases in the numbers of elderly in society [2], has provided not only an opportunity for the deployment of technology in home based healthcare, but also a real need for successful solutions to meet end user demands. Many efforts have been directed towards the development of new and emerging technologies to further support the person in their own home for example fall detectors [110], behavioural analysis systems [115], intelligent garments [5, 97] to name but a few. Nevertheless, the real underlying need still exists in the development of an infrastructure which will support the delivery and management of the healthcare coupled with a realistic re-imbursement scheme. An additional requirement is the consideration which should be given to existing healthcare practice and how the introduction of new paradigms will impact upon this. This implicitly will require the support from both public and private healthcare providers to be ﬂexible in the adoption of new solutions in addition to promoting their use. 104 HomeCI - A visual editor for healthcare professionals in the design of home based care

In this paper we attempt to address the issue of how technology within the home environments can be managed by healthcare professionals. Speciﬁcally we attempt to address the problem of supporting healthcare professionals who wish to specify in a non-technical manner how the technology within the home should be monitored and what constitutes an abnormal situation. In the following Section we provide a more general overview to the Background to our study. In the Methods Section we describe the development of a visual editor, referred to as HomeCI, to support the design of home based healthcare rules in a non technical manner and in the Results Section we describe the evaluation of a prototype version of the HomeCI system. In the Discussions and Conclusions we discuss the initial impact of this concept following evaluation and explain how this work integrates with a larger study in the development of information management within smart environments.

8.2 Background

Within the realms of home healthcare delivery terms such as service model or care model are commonly used as a means to represent the infrastructure through which the care is managed and delivered. Such an infrastructure can therefore be considered to act as a means of defining all of the stakeholders in the care provision process (including the patient) and how they interact with one another, in addition to defining and managing the devices and services and specifying how the care will actually be delivered [94, 105]. Although a vast amount of work has been focused towards the development of the home based technology, there is a deficit in terms of the amount of effort and indeed understanding which has been gained relating to the requirements of the healthcare operators of such systems. Within this paradigm healthcare operators may be from a number of professional backgrounds for example occupational therapy, physiotherapy, pharmacy, nursing and general medicine. All may have the responsibility to define the necessary requirements which will specify what type of technology should be included within the home environment and under which conditions should patterns of behaviour be considered as normal or abnormal. In addition to this, healthcare professionals may also be required to provide a form of intervention once instances of abnormal behaviour or alarm situations have actually been identified. The current study aims to address needs of healthcare professionals who manage the deployment of technology to support independent living. Specifically, our efforts have been directed towards the development of an interface for healthcare professionals to allow them to establish the conditions and rules which will govern and monitor technology to be deployed in a person’s home. The approach is based on the design of a user interface which operates through the usage of visual notations. The premise of this approach aims to exploit the fact that the designer of the rules i.e. the healthcare professional will have limited knowledge of the backend processing of the monitoring system and will be largely from a non technical background.

8.3 Methods

To undertake the proposed study required an assessment of typical situations which healthcare professionals would likely be faced with. In the second instance, it was required to develop HomeCI - A visual editor for healthcare professionals in the design of home based care 105 a prototype tool (HomeCI) which would act as the interface between the healthcare provider and the care model. The following Sections elaborate upon these concepts in addition to providing the methodology adopted for the evaluation of the HomeCI prototype.

8.3.1 Establishment of scenarios

To facilitate the design of a suitable interface a number of typical scenarios requiring home based support were established. Figure 8.1 summarises the ﬁve scenarios considered within the study.

Scenario Summary Cooking The cooker should not be switched on until the person has added some food to a pot and placed a pot on the cooker. The cooker should not be switched on for more than 1 hour. Dressing Once a person wakes and gets out of bed, within a specified time (30 minutes) it is normal that they first go to the bathroom and wash and then proceed to dress themselves. Both of these activities should be performed before the person leaves the house. Grooming Once in the bathroom the person should wash, brush their teeth and comb their hair. There is no desired sequence to these activities, however, the duration within which they should be completed would not normally exceed 30 minutes. Medication A person must take their medication within 1 hour after getting out of bed. Within this period management they must also ensure that they have had something to eat prior to taking the medication. Drink The preparation of a drink should follow a general sequence of steps. For example, the kettle Preparation should not be boiled unless water has been added and the person should not consider the activity finished until they have added some milk or flavouring, tea/coffee and the boiled water to a cup.

Figure 8.1: Scenarios to be used within study to assist in the initial design of the HomeCI user interface tool

The scenarios were established in conjunction with a geriatric consultant, experienced in assessment of the requirements of patients living in their own homes in addition to having an appreciation of how technology could be deployed to support independent living [29]. All scenarios are based on the premise that the home environment will be highly sensorised hence the person’s interaction with the environment and their movements can all be monitored.

8.3.2 Development of HomeCI user interface

For the purposes of the prototype only the scenarios as in Figure 8.1 have been considered. The HomeCI visual editor is based on the usage of visual notations. Figure 8.2 shows the initial constructs of the visual notation used. To avoid complexity in the initial version of the HomeCI prototype only sequences and series of events in addition to timing constraints of operations have been included. Increased complexity within the control structures will be addressed in the following version of the prototype. The underlying computation of the system is therefore required to map the visual notation onto a form of computational logic which can then be converted into a computer language. Based on the control structures as presented in Figure 8.2 it has been possible to address this issue through the representation of the computational logic in terms of pseudo 106 HomeCI - A visual editor for healthcare professionals in the design of home based care

Event Event e.g. turn on tap

Direction of flow of events

Sequence: Event B follows A B Event A

Series: Events A, B and C A ++B C must all take place, however, there is no expected order

Time Constraint: All series/ A B C sequences of events must take place within a specified time.

Figure 8.2: Graphical representation of visual notation within HomeCI user interface code. The pseudo code included in the example in Figure 8.3 provides an indication of how the HomeCI visual notation represented by the sequence event A (last event) followed by event B (event) can be presented. In this example if event B does not follow event A an alarm is raised (ACTIVATE warning) otherwise the system proceeds without warning.

BEGIN WHILE system operational REPEAT WAIT on event UNTIL sensor activity IF event AFTER last event THEN SET last event equal to event ELSE ACTIVATE warning ENDIF ENDWHILE END

Figure 8.3: Pseudo code for how HomeCI visual notaction can be presented

An example of the HomeCI visual editor developed within the study is shown in Figure 8.4. The visual editor was created using Macromedia Flash. This supported the development of a platform which could be easily deployed either on a mainstream operating system as a standalone application or alternatively deployed through a web browser. On the left hand side of the editor is a tool bar where users can select the events they wish to include in the design of the condition to be monitored. These events include for example ’turn on tap’ or ’get out of bed’. All events are represented by icons which are representative of their meaning and in addition have a small textual description. Along the bottom of the editor are the necessary symbols to support the construction of the HomeCI notation (as presented in Figure 8.2). Both the events and the symbols can be dragged onto the main HomeCI - A visual editor for healthcare professionals in the design of home based care 107 canvas of the application and can be freely moved around. For the purposes of evaluation a timer has been added to record the time taken to design one complete rule. The example presented in Figure 8.4 is representative of the ’Dressing’ scenario as previously described in Figure 8.1.

Figure 8.4: Screen Shot of HomeCI visual editor

8.3.3 Evaluation approach for HomeCI prototype

To evaluate the HomeCI prototype a limited usability test with 4 users was designed. Two of the users were selected from Northern Ireland and two were selected from Northern Sweden. All users were experienced in working in the domain of elderly care and had good appreciation of the needs of technological solutions to be deployed in the home environment to support independent living. The professional backgrounds of the users were occupational therapy, registered nursing and physiotherapy. None of the users involved in the evaluation had ever written a computer program, nor were they familiar with general computer programming principles. All users were familiar with general usage of a PC and the Internet. Prior to using the system the users were given a short presentation detailing the goals of the HomeCI concept which was followed by an explanation of the constructs used to 108 HomeCI - A visual editor for healthcare professionals in the design of home based care build the rules with the visual notation. This was then followed by a brief demonstration of the HomeCI visual editor. This portion of the evaluation lasted for approximately 20 minutes. The evaluation was based on the scenarios as presented in Figure 8.1 and structured as follows:

• Scenario 1 (Cooking): This example was shown to the users. The solution was constructed using HomeCI by the investigator who was conducting the evaluation. • Scenario 2 (Dressing): Users were shown the ﬁnal solution on the HomeCI editor and were shown how to revert back to the narrative from the visual notation. • Scenario 3 (Grooming): Users were given the narrative and asked to create the rules using the HomeCI tool. • Scenario 4 (Medication Management): Users were given the narrative and asked to create the rules using the HomeCI tool. • Scenario 5 (Drink Preparation): Users were given the solution on the HomeCI editor and asked to create the narrative.

The users were not given any feedback or support during the evaluation. Upon completion of the evaluation the users where shown the desired results for Scenarios 3-5 and were then asked to complete a brief questionnaire.

1 2 3 4 Avg. Scenario 3 Icons 3 3 3 3 3 Notation 0 1 1 2 1 Time 2 2 1 2 1.75 Total (%) 71.4 85.7 71.4 100 82.1 Time (s) 156 169 141 105 142.8 Scenario 4 1 2 3 4 Avg. Icons 3 3 3 3 3 Overall 8 8 7 8 7.75 experience Notation 2 2 2 2 2 Ease of using 8 8 9 9 8.5 Time 2 1 2 2 1.75 icons Ease of use of 9 10 9 8 9 Total (%) 100 85.7 100 100 96.4 interface Time (s) 87 100 113 119 104.7 Ease to produce 8 9 5 9 7.75 Scenario 5 narrative Usefulness of 9 10 7 9 8.75 Narrative (%) 75 95 85 90 86.2 system

(a) Assessment of user interaction with HomeCI (b) User’s experience in the evaluation of the interface HomecI prototype

Figure 8.5: Results from evaluation of the HomeCI prototype HomeCI - A visual editor for healthcare professionals in the design of home based care 109

8.4 Results

Figure 8.5(a) presents the results from the evaluation for the 4 users involved. The user re- sponses to Scenarios 3,4 and 5 were each assessed following the investigation. For Scenarios 3 and 4 each solution was assessed on 3 criteria; correct use of icons (maximum 3 marks), correct use of notation (maximum 2 marks) and correct insertion of timing constraints (maximum 2 marks). The time to complete both Scenarios 3 and 4 was also recorded. Overall this may be considered as a rudimentary assessment scheme, nevertheless, it can be considered sufﬁcient to assess the current version of the system. (Assessment of more complex scenarios would require a more elaborate marking scheme.) For Scenario 5 the narrative produced was assessed and scored as a percentage. As can be seen from Figure 8.5(a) the average percentage scores for all Scenarios were between 82.1% and 96.4%. All users achieved either an equal or improved result in Scenario 4 in comparison to Scenario 3 and in addition a reduced amount of time required to complete the task was recorded in 3 out of the 4 users. Following the evaluation the users were asked a number of questions regarding the HomeCI interface and asked to score their response between 1 and 10 (1 - poor, 10 - excellent). The results from the users in terms of their experience of using the tool can be found in Figure 8.5(b).

8.5 Discussion

The design of the HomeCI editor has been based around the use of visual notations to avoid the technical complexities in establishing logic based rules. The HomeCI visual editor was evaluated by 4 healthcare professionals. Following a limited amount of training on this system all 4 users were able to construct two Scenarios and were also able to interpret and produce a narrative for a previously prepared solution. The performance of the users to develop the solutions ranged between 71.4% and 100%. The main errors noted related to the incor- rect usage of the notation. Following the evaluation users felt that they were not given enough time to learn these concepts, nevertheless, the performance achieved was still relatively high. In addition, the performance in producing a narrative based on a prepared solution had an average of 86.2%. When the users were asked to evaluate their experience of using the tool all provided extremely positive feedback. Based on the series of questions asked (Figure 8.5(b)) average scores of 7.75-9 out of 10 were achieved. All users reported a very high level of satisfaction in terms of being able to both use the system and understand the icons. In addition, all of the users felt that the general concept of HomeCI would be useful for home based care support. This feedback in a way validates the use of the HomeCI visual editor and has demonstrated how it is possible to use visual notation to allow those with a non-technical background to manage the deployment of technology within the home environment. 110 HomeCI - A visual editor for healthcare professionals in the design of home based care

8.6 Conclusions

Within this study we have attempted to address the development of a user interface to support healthcare professionals in the design of managing technology which is to be deployed within a person’s home. Such a tool has the potential to offer healthcare professionals a greater level of autonomy when using care management systems and reduce their reliance on technical support. Given the success of the initial usability study we intend to further develop the concepts of the HomeCI visual editor. In the ﬁrst instance we intend to increase the functionality of the HomeCI notation in terms of the constructs it can represent. In conjunction with these proposed developments we intend to develop the underlying system which can parse the rules generated in the HomeCI prototype, and generate rules which can subsequently be easily deployed into a smart home. Consideration will also be directed towards the impact HomeCI will have on the daily work of the care providers and which organizational issues should be taken into account. Finally, the next stages of evaluation will include a larger number of users with a wider range of backgrounds.

8.7 Acknowledgement

The authors wish to express their thanks for the 4 users who evaluated the HomeCI prototype. The comments received following evaluation were extremely useful. Part 9

Assessing the Impact of Individual Sensor Reliability Within Smart Living Environments

Citation: Chris Nugent, Xin Hong, Josef Hallberg, Dewar Finlay, Kåre Synnes, "Assessing the Impact of Individual Sensor Reliability Within Smart Living Environments",Inthe 4th Annual IEEE Conference on Automation Science and Engineering (CASE), pages 685–690, Wash- ington DC, USA, August 2008.

Contributions: This article was achieved together with Chris Nugent, Xin Hong, De- war Finlay, and Kåre Synnes. Chris Nugent was the main author and also initiator of this work, however, Xin Hong developed the computational models and performed the data analysis. My main contribution was the creation of the sensor platform which was used for the collection and management of sensor data, and used to perform the tests which were a signiﬁcant practical part of this paper. I also assisted in the design and planning of the experiment to a certain extent.

111

Assessing the Impact of Individual Sensor Reliability Within Smart Living Environments 113

Assessing the Impact of Individual Sensor Reliability Within Smart Living Environments

Chris Nugent1, Xin Hong1, Josef Hallberg2, Dewar Finlay1, Kåre Synnes2, 1 School of Computing and Mathematics, University of Ulster, N. Ireland 2 Luleå University of Technology, Sweden

Abstract

The potential of smart living environments to provide a form of independent living for the ageing population is becoming more recognised. These environments are comprised of sensors which are used to assess the state of the environment, some form of information management to process the sensor data and ﬁnally a suite of actuators which can be used to change the state of the environment. When providing a form of support which may impinge upon the wellbeing of the end user it is essential that a high degree of reliability can be maintained. Within this paper we present an information management framework to process sensor based data within smart environments. Based on this framework we assess the impact of sensor reliability on the classiﬁcation of activities of daily living. From this assessment we show how it is possible to identify which sensors within a given set of experiments can be considered to be the most critical and as such consider how this information may be used for managing sensor reliability from a practical point of view.

9.1 Introduction

As a result of the predicted growth of the population it is expected that the number of elderly will reach 2 billion by the year 2050 [2]. This is in comparison to 600 million in the year 2000. Key areas to be addressed within this cohort include management of long term health conditions, support with activities of daily living and increased levels of social par- ticipation. The desired goal can be considered as ﬁnding solutions which can address these issues and increase the amount of time a person can remain in their own home prior to a form of institutionalised based care provision. Assistive technologies and smart environments are becoming recognised as ways of addressing some of the problems associated with the growing size of the population [98]. Previ- ous efforts in this area have been directed towards the establishment of systems which provide a form of preventative care delivery as opposed to reactive care delivery [99]. This has the desired effect of identifying problems at an early stage and intervening prior to the onset of a more serious condition. At the core of all of these systems are sensor based technologies which have the ability to record information about the user and their interaction with the environment. Given that 114 Assessing the Impact of Individual Sensor Reliability Within Smart Living Environments these provide the basic pieces of information upon which all further processing is based, their integrity and reliability is of utmost importance. It has been the aim of the current study to assess the impact of sensor reliability on the overall process of monitoring activities of daily living within smart environments. Within this work we have aimed to identify the impact of sensor failure on the overall decision making process. In addition, we have investigated how this information may be used from a practical deployment perspective. The remainder of the paper is structured as follows. Section 9.2 presents an overview of work in the area of assistive technologies and smart environments. Section 9.3 presents the Methods of the study and details of the information processing infrastructure. Results are presented in Section 9.4 and Section 9.5 presents Conclusions and scope for further work.

9.2 Background

Assistive technologies are considered to be any form of device or service which can offer improved levels of independence for the elderly or for those with a form of disability. Assistive technologies range from walking aids and bath rails to medication management devices and automatic vacuum cleaners [98]. At a more general level smart living environments have the ability to monitor the behaviour of the person within the environment, both in terms of their physical status and which objects within the environment they are interacting with. As a result of this monitoring the environment can be adapted to meet the perceived needs of the person [28]. Sensor technology can range from vital sign devices such as blood pressure monitors, heart rate monitors and devices which can measure body temperature to sensors which can detect presence in a room or detect a door being opened. Once all of the data has been recorded it is then necessary for data processing to take place to identify if the person requires a form of assistance. As previously mentioned, one of the key elements of the information processing is to adopt a form of preventative care support. This requires a degree of intelligence which should take into consideration the current state of the person’s environment, along with person speciﬁc information and as a result identify any perceived issues requiring a form of intervention [28, 115]. Based on the output of this processing the person’s environment should be adapted accordingly through a suite of actuators. Actuators can be used for example to open or close doors or windows, vary the ambient temperature or control the delivery of various multimodal messages. One of the key elements associated with the latter form of multi-modal user interaction, especially for the elderly population, is identiﬁcation of the correct manner in which this information can be relayed to the user. Figure 9.1 aims to encapsulate the various processes within a smart environment ranging from the understanding of user needs through to how feedback should be presented in the most effective manner. Activities of Daily Living (ADL) represent a key set of activities which may be monitored within the home environment [112]. These can be used as metrics to assess a person’s wellbeing and in certain circumstances can be used to measure both cognitive and physical decline. Typical activities of interest include preparing a meal or a drink, using the telephone, managing medication intake, to name but a few. From a technological perspective it is possible Assessing the Impact of Individual Sensor Reliability Within Smart Living Environments 115

Measurements of factors associated with age related impairments – requires further clinical input

y b Sensor Meas d urements e D

n e r n m e o v i o t t o n a h g s c e i t n f r i St i a s a t a s d i

o m P t a s e l e n o t

t - C r o D n o

a o f l f

F e -

c m D t a C e h

I e e d l m e

R r a v b s -

i a a

o t

r e a a r n

t i c

P g p n i

n A g m g I

Anticipated Impact: delay of Continual Feedback to onset/pogression of age related Person impairment Investigation of Personalsied HCI for mobile device to provide feedback to compliance to personal eInclusion prescription

Figure 9.1: The various stages involved in the provision of independent living within smart living environments. to introduce technology into the home environment which would have the ability monitor the person and hence assess their ability to undertake the aforementioned ADLs. Nevertheless, issues surrounding personal privacy, technical installations, costs of technology along with the maintenance of the technology limit, to a certain extent, what can be deployed in practical terms. Taking these issues into consideration has resulted in the widespread use within smart living environments of anonymous binary sensors. Binary sensors do not have the ability to directly identify people and can only present two possible values as outputs namely ’0’ or ’1’. Typical examples of binary sensors deployed within smart environments include pressure mats, contact switches and movement detectors. A number of studies reporting the use of binary and related sensors have been undertaken for the purposes of activity recognition [112, 151, 88, 143]. Nevertheless, sensor data can be considered to be highly dynamic and prone to noise and errors [117]. For example, a pressure mat installed at the side of a person’s bed may have an intermittent technical problem and as a result a zero reading may not necessarily mean that the person has not stepped on the mat after leaving the bed. It may in fact be the case that the person has stepped on the mat following leaving the bed, however, due to some malfunction the desired activation of the pressure mat sensor has not been reported. In addition, many sensors are wireless to further improve deployability. Nevertheless, the use of wireless sensors adds another dimension of sensor failure i.e. not being able to deliver the message to the receiver due to loss of data during transmission. To date there has been little exploration within the area of smart living environments in 116 Assessing the Impact of Individual Sensor Reliability Within Smart Living Environments terms of the notion of ’trust’ of the proposed activity inference processing systems and the overall impact which may be caused as a result of sensor failure. Techniques which can therefore provide a way of assessing sensor uncertainty may therefore be viewed as offering potential solutions to this problem. In our current work we have introduced a technique of activity inferencing which offers a proposed solution to the problems of management of sensor uncertainty [62]. In the following Sections we investigate the impact of individual sensor reliability on the overall inference process of ADL assessment.

9.3 Methods

Our current work has focused on the ADL of preparing a simple drink. In the following Sec- tions we will present details of the environment within which our work has been conducted along with the steps necessary to assess the ADL of drink preparation. This will be followed by an overview of the information management processing infrastructure used within our current work along with details of the planned assessments of how sensor failure will impact upon the overall decision making process.

9.3.1 Smart environment

To assess if a person is preparing a drink in addition to identifying which drink is being prepared requires embedding a number of sensors into the smart environment. The sensors used within this experiment were wireless binary sensors with reed switch technology to facilitate triggering (Figure 9.2). Once triggered the sensor sends its message nine times (to maximise the probability of the message being received) to the sensor receiver. The sensor receiver is queried by the information processing module for sensor updates every 0.5 seconds. Figure 9.3 provides the details of the sensors used within our current experiments to differentiate between a hot drink (tea or coffee) and a cold drink. Within the hot drink category it can be seen that some of the sensors are optional. For example, not every person will put sugar into their drink or the kettle may not require ﬁlling with water each time a drink is prepared.

9.3.2 Information management

To provide a suitable framework to manage all of the information generated from the aforementioned suite of sensors we have established a series of evidential networks to infer activities of making a hot drink or making a cold drink. These networks have been based on a hybrid mix of methods capable of managing information and its potential uncertainty. The overall process of activity inferencing within this scheme may be represented as a set of 7 possible steps starting from the point of sensor activation through to the ﬁnal stages of combination of all sources of evidence available to make the ﬁnal decision [62]. Figure 9.4 presents the evidential networks of making a cold drink and making a hot drink. Assessing the Impact of Individual Sensor Reliability Within Smart Living Environments 117

(a) Picture of smart kitchen environment

(b) Cupboard with door sensor (c) Kettle with tilt switch and contact switch on tap

(d) Contact sensors on sugar, tea (e) Contact sensor on coffee in ’on’ and coffee state

Figure 9.2: Sensors within smart laboratory environment to assess the ADL of preparing a drink. 118 Assessing the Impact of Individual Sensor Reliability Within Smart Living Environments

Sensor Description Tea Coffee Cold Name Drink Fridge Used to detect the opening and closing of the O O Y Sensor fridge Cupboard Used to detect a cup or glass being removed Y Y Y Sensor from the cupboard Coffee Used to detect if coffee was taken from the N Y N Sensor cupboard Tea Used to detect if tea was taken from the Y N N Sensor cupboard Sugar Used to detect if sugar was taken from the O O N Sensor cupboard Water Detects if the taps on the sink are turned on. O O N sensor Kettle Detects if water is poured from the kettle. Y Y N sensor

Figure 9.3: Summary of sensor technology used to analyse the ADL of preparing a simple drink (O - Optional, Y - Yes, N - No).

In terms of the required processing steps, the lowest level in the evidential network is where evidence appears on a sensor node (for example scup). This can then be associated with an object context for example cup. This information may then be summed onto a composite node through the use of an equally weighted sum operation [83]. Alternatively the object context may be used to deduce objects in more detail. At the activity node, outputs from the composite context nodes along with object nodes can be combined by Dempster’s combination rule [132]. (For full details refer to [62].) Over a period of 4 weeks the sensor based system was tested on 58 instances of people preparing either a cold drink or a hot dink (either tea or coffee). The result following analysis of the initial data have validated the conceptual approach and have shown the ability of the evidential networks to correctly classify 100% of all of the drink preparation experiments [61].

9.3.3 Assessment of sensor reliability

With an appropriate smart environment in place along with a theoretical framework for information processing, the focus of the current study was directed towards assessing the impact of sensor reliability on the overall classiﬁcation of ADLs. (In the aforementioned 58 experiments the issue of sensor reliability was not taken into consideration.) The entire concept of the evidential networks has been based on the Dempster-Shafer (DS) theory of evidence [132]. DS theory is a generalization of traditional probability which allows us to better quantify uncertainty. At the core of the theory is the proposition of the frame of discernment. Assessing the Impact of Individual Sensor Reliability Within Smart Living Environments 119

Make cold drink Make hot drink

cup, juice cup, water, kettle, tea/coffee milk sugar

tea / ssug cup juice fridge cup water kettle coffee

scup sfri scup swat sket fridge

tea coffee sfri

stea scof

Node Context Link Relation

sensor A is associated with object B sensor A B sensor

object A derives object B A B object object (associated with a sensor) A A and B are compulsory to C; A, B and C object (derived from can be objects or activities object B other object) A A and B are alternative to C; A, B and C compound object (a set of can be objects or activities B object compulsary objects) A is compulsory to activity B; A can be an A B activity activity object, a compound object, or an activity

A is optional to activity B; A can be an A B object, or an activity

Figure 9.4: Examples of evidential networks for making a cold drink and making a hot drink along with a legend to describe the notation used [62]. 120 Assessing the Impact of Individual Sensor Reliability Within Smart Living Environments

When considering simple anonymous binary sensors each sensor may only take the value of ’1’ to indicate active and ’0’ to indicate inactive. These two values comprise the exhaustive set of mutually exclusive values that the sensor can hold. In DS theory this is referred to as the frame of discernment of the sensor and is denoted by Θ. For example, consider the sensor sket which represents the kettle sensor. The frame of discernment may be expressed as {sket, ¬sket} where sket represents the sensor being active and ¬sket represents the sensor being inactive. Once the frame of discernment is established it is then possible with DS theory to assign a degree of belief in the observation. The distribution of a unit of belief over the frame of discernment is called evidence and is a number in the range [0,1]. A function m :2Θ → [0,1] is called a mass function and represents the distribution of belief. This distribution of belief must also satisfy the following two conditions:

(1) m(θ)=0 Θ : the emptyset (2) Σ A⊆ Θ m(A) = 1 A: a subset of Θ

With these propositions a mass value can be committed to either a singleton or a subset of Θ. It is this distinct feature of the DS theory which makes it more expressive than traditional probability theory. When there is insufficient evidence to distribute belief to individual elements in the subset a mass value will be committed to the subset. The output of the model is presented in terms of a belief (Bel) value. The value of Bel represents the degree of belief to which the evidence supports a given proposition. One of the key evidential operations which may be deployed within DS theory, which is of direct relevance to the proposed evidential networks for ADL assessment, is that of discounting [77]. (The first stage of the evidential network as shown in Figure 9.4 allows for the sensor evidence to be discounted.) Discounting allows for practical issues surrounding sensor reliability to be taken into account. As such the impact of evidence may be discounted in an effort to reflect the reliability of the sensor. The discounting rate r may be expressed as r(0 ≤ r ≤ 1). Taking into account the notion of discounting, the discounted mass function can be presented as: (1 − r)m(A)A ⊂ Θ mr(A)= r +(1 − r)m(Θ)A = Θ

where, (a) r=0 the source is absolutely reliable (b) 0

Given that the DS theory provides the ability to take into account reliability and this may be reﬂected in the sensor node values of the proposed evidential networks the current study investigated the impact of the evidential operation of discounting on the overall ADL inferencing process. Within our experiments we varied the value of the discounting rate for sensors in the range [0,1] at intervals of 0.2. This aimed to provide a general insight into the effects of sensor reliability between the two boundaries of totally reliable and totally Assessing the Impact of Individual Sensor Reliability Within Smart Living Environments 121 unreliable. Under real deployment of sensor based systems the discounting rate would be based on factors such as, for example, how many times had the sensor failed in the past and what was the likelihood that there would be failure due to loss of battery power.

9.4 Results

Figure 9.5 presents the output of the evidential network with each sensor being considered to be in failure on a case-per-case basis. Here sensor failure is considered to be when the sensor’s activation has not been detected and hence it is not possible to say whether it has actually been activated or not. The output of the evidential network for either making a hot drink or making a cold drink is presented in terms of a belief (Bel) value. The upper trace in Figure 9.5 represents the Bel value for a hot drink and the lower trace represents the Bel value for a cold drink. With information available for all sensors the argument for a hot drink becomes more compelling than cold drink hence the difference in the Bel values between the two tasks. Most notable is the instance of cup sensor failure. With this sensor in failure there is little evidence to support making a cold drink (one of the two core sensors are now in failure), hence the reason of the low Bel value in this instance. These results indicate that for the evidential network for assessing the ADL of a hot drink that the sensors on the cup and tea have the largest impact on the overall process. This is related to the fact that they are mandatory in the process of making a hot drink (preparing tea) whilst other sensors are either optional or not required at all (refer to Figure 9.3). Within the evidential networks this information therefore provides a signiﬁcant contribution to the overall ﬁnal value of belief assigned. It is for this reason that these sensors have been examined further in the following experiments.

1 0.9 0.8 0.7 0.6 0.5 Bel 0.4 0.3 0.2 0.1 0 None wat fri sug ket cup tea Failure_sensors combination

Figure 9.5: Bel values for hot drink (upper trace) and cold drink (lower trace) with each of the sensors considered to be in a state of failure.

Figure 9.6 depicts the effect on the overall Bel value for making a hot drink when the value for r was varied in the range [0,1] for the tea sensor, the fridge sensor and the cup sensor. In Figure 9.7 results are presented in the scenario when it is considered that both the cup sensor and the tea sensor have both a degree of uncertainty at the same time. Figure 122 Assessing the Impact of Individual Sensor Reliability Within Smart Living Environments

9.8 depicts the effects on the Bel value when both the reliability of the fridge sensor and tea sensor are taken into consideration. (Note: in this ﬁgure all results from the 6 discounting values are superimposed on each other due to achieving similar values hence only one line is visible.) Results presented in Figure 9.5 demonstrate which sensors in the evidential network have the greatest impact on the overall result during instances of failure. In this case it was found that the cup sensor and tea sensor had the largest impact, in terms of reliability of the model, following their failure. Taking this into consideration the remainder of the experiments considered the degree to which these levels of reliability would impinge upon the overall result. The impact of sensor reliability on the overall Bel value for the target ADL is rather complex. The result depends on many factors for example, how many of the sensors under examination are considered to be in failure, which value should be used as the discounting rate for sensor reliability and what are the combinations of sensor failure which may have the largest impact.

Tea sensor failure Fridge sensor failure Cup sensor failure

1.00 ) 0.98 0.96 0.94 0.92 0.90 0.88 Bel(Making hot drink hot Bel(Making 0.86 0.0 0.2 0.4 0.6 0.8 1.0 Reliability discount rate

Figure 9.6: Effects on the Bel value for making a hot drink following varying the value of r on the sensor values of tea sensor, fridge sensor and cup sensor.

Figure 9.5 shows that when only one sensor has failed our inference model is credible to distinguish ’Making hot drink’ as opposed to suggesting ’Making cold drink’ based on the Bel value produced. Based on this result we can see that the sensors tea and cup have the most signiﬁcant impact on the Bel value for the ’Making hot drink’ ADL. Also, the fridge sensor (which is an optional sensor in detecting the activity) has a lesser impact. Based on this initial analysis we then further examined how the sensor reliability (and the discounting rate) would effect the Bel value on ’Making hot drink’ when individual sensors tea, cup,or fri or the combination of tea and cup and tea and fri were considered to be in failure. Figure 9.6 shows the impact on the Bel value for the decision of ’Making hot drink’ when the discounting rate of the sensor tea, cup,orfri was varied from 0 to 1 when all other remaining sensors were considered to be operating correctly. The sensor tea and cup are considered to be compulsory when performing the activity ’Making hot drink’, especially when preparing tea. Hence, when they are not active evidence is accumulated against the activity being performed. Nevertheless, when their discounting rate increases, as shown in Figure 9.6, the evidence provides less contribution to the average sum of beliefs related to Assessing the Impact of Individual Sensor Reliability Within Smart Living Environments 123

r(Cup)=0 r(Cup)=0.2 r(Cup)=0.4 r(Cup)=0.6 r(Cup)=0.8 r(Cup)=1.0 0.82

) 0.8 0.78 0.76 0.74 0.72 0.7 0.68 0.66 Bel(Making hot drink hot Bel(Making 0.64 0.62 0 0.2 0.4 0.6 0.8 1 Reliability discount rate (Tea sensor)

Figure 9.7: Effects on the Bel value for making a hot drink when both the reliability of the tea sensor and cup sensor are taken into consideration.

r(Fridge)=0 r(Fridg)=0.2 r(Fridge)=0.4 r(Fridge)=0.6 r(Fridge)=0.8 r(Fridge)=1

) 0.775 0.77 0.765 0.76 0.755

aking hot drink 0.75 0.745

Bel ( M 0.74 0 0.2 0.4 0.6 0.8 1 Reliability discount rate (Tea)

Figure 9.8: Effects on the Bel value for making a hot drink when both the reliability of the tea sensor and fridge sensor are taken into consideration. the activity. This consequently results in increasing the belief on the activity of ’Making hot drink’. On the other hand, the fridge sensor may be considered as optional in this scenario. When all compulsory sensors are fully active there is already a high degree of belief about the activity of ’Making hot drink’. The evidence from the fridge sensor contributes little to support or refute the overall activity decision. This is clearly reflected in Figure 9.6 where the variance of the discounting rate is shown not to impinge upon the Bel value for the activity. Figure 9.7 and 9.8 allow us to examine the impact of different combinations of sensor failure. These graphs show the belief changes when two sensors have failed with discounting rates being varied from 0 to 1. These results prove that the compulsory sensors (e.g. tea, cup) in comparison to the optional sensors have a bigger impact on the activity decision making process. In addition their reliability discounting rate will have an affect on the belief of the overall decision in contrast with the optional sensors (e.g. fri) which have a lesser impact. From these results we can therefore state that depending on the sensor’s fundamental role in 124 Assessing the Impact of Individual Sensor Reliability Within Smart Living Environments the overall decision making process (i.e. compulsory or optional) varying degrees of impact on the belief of the final decision can be identified. Also, it is possible with this approach to identify which sensor will have the biggest impact in the overall decision making process and as such quantify the effect of this impact. This information can then be used from a practical point of view to identify key sensors which should be given regular attention in terms of assessing their correct functionality.

9.5 Conclusions

Smart living environments provide a realistic solution to some of the problems which can be expected as a result of the predicted growth of the population. At the centre of these environments are sensor technologies which record information about the person and how they are interacting in the environment. In the current study we have investigated the impact of sensor reliability on the overall process of monitoring ADLs. Based on this work we have been able to show that it is possible to identify which sensors, if considered to be in failure, will have the largest impact on the overall inferencing process. These results are useful from a practical point of view and can be used to ensure that in real situations of sensor deployment identiﬁcation of sensors which may require regular checking can be made.

9.6 Acknowledgements

This work has been partly supported by the Nestling Technologies Project and the Centre for Distance-Spanning Healthcare (CDH). Part 10

Creating Dynamic Groups using Context-awareness

Citation: Josef Hallberg, Mia Backlund-Norberg, Johan Kristiansson, Kåre Synnes, Chris Nugent, "Creating Dynamic Groups using Context- awareness",Inthe 6th International ACM Conference on Mobile and Ubiquitous Multimedia (MUM), pages 42–49, Oulu, Finland, Decem- ber 2007.

Contributions: I am the main author and also the initiator of this research. Neverthelss, the article was written in collaboration with mainly Mia Backlund- Norberg and also Johan Kristiansson, Kåre Synnes, and Chris Nugent. Credit should also be given to the creators of the openHome Suite, as dynamic groups utilize several of those tools for its functionality.

125

Creating Dynamic Groups using Context-awareness 127

Creating Dynamic Groups using Context-awareness

Josef Hallberg1, Mia Backlund-Norbergg1, Johan Kristiansson2 Kåre Synnes1, Chris Nugent3 1 Luleå University of Technology, Sweden {josef.hallberg, mia.backlund-norberg, kare.synnes}@ltu.se 2 Ericsson Research, Luleå, Sweden [email protected] 3 School of Computing and Mathematics, University of Ulster, N. Ireland [email protected]

Abstract

This article presents the conceptual communication model of dynamic groups, that dynamically utilizes three traditional communication metaphors through the use of context-based information. Dynamic groups makes creation, management and usage of groups easy. It enables social network structures to be maintained in both virtual and face-to-face settings as well as in the combination thereof. This article deﬁnes the dynamic management of advanced contact lists which can include presence and status information, a/synchronous multimedia communication tools, and methods for structuring social networks. It also contains an initial evaluation and a proposed architecture for technichal realisation.

10.1 Introduction

It is no longer self-evident that communication should take place face-to-face. Many choose to maintain virtual social networks, through the use of internet tools, as well as their normal social networks with friends, colleagues, etc. The availability that the Internet provides has made it possible to find and communicate with people all over the world [32]. While these internet tools allow people to meet and share experience in new ways, they are not always easy to use, especially not in mobile terminals with limited input capabilities. Using them can also result in an increased amount of management and stress. This is particularly true for conferencing communication tools or the next-generation multimedia telephony system [38] where users have to manually invite all participating users to establish a session. For example, users are typically required to remember multiple addresses like SIP URIs or phone numbers to other users, or alternatively manually selecting them from a personal friends list, assuming the users can be found there. This paper outlines a model based on context-awareness to aid setting up group communication sessions. Imagine a communication system where a user can form groups effortlessly and on the user’s terms with minimal user interaction. Groups can be formed whenever and 128 Creating Dynamic Groups using Context-awareness wherever the user chooses, be it for over a day, or for a longer period of time. Using the proposed model, a dynamic group can either be formed manually between people who are co-located, by joining a desired group, or by a user who invites others (either by searching, or picking from a list), or automatically based on predefined context-aware rules. The authors believe that dynamic groups can greatly simplify the way people communicate and cooperate in groups. The main focus of this article is to define the dynamic group model (e.g. how to find users, form and maintain groups, and support context sharing) and not go into details about network and network protocols. There are a number of research questions related to dynamic groups:

• How should dynamic groups be formed and maintained? • How should sensor-data and contexts be stored to support the functionality of dynamic groups?

• How should rules for automatic establishment of dynamic groups be created? • What should a proﬁle system contain to provide the functionality of the dynamic groups?

To illustrate the advantages of dynamic groups we present two different scenarios. One scenario is from a healthcare perspective where the priority for simplicity and automaticity motivate the use of smart environments in conjunction with decision support rules for creating dynamic groups. The other is from a normal every-day-use perspective where a user simply might want to uphold his/her virtual social network in a simple manner.

10.1.1 Scenario 1 - Healthcare

Hulda is 77 years old and is still living at home. She uses the dynamic group system in order to maintain social relations. With this tool Hulda has easy access to her friends, family, and healthcare professionals (see Figure 10.1). In addition to communication, the system which Hulda uses has some context-aware rules built in for medical reasons, as well as calendar functions to remind her of her social events. Hulda always start her day with a cup of tea in her living room. She has requested that the dynamic groups system automatically connects her to others who are currently having their morning tea or breakfast. The living-room sofa is equipped with sensors and detect when Hulda sits down, and a query to the proﬁle database for "current activity: breakfast" and/or any public group with the keyword "breakfast" is issued and Hulda gets connected. After Hulda has had her tea she prepares for her morning exercise. Her smart clothing is equipped with a number of health monitoring sensors, which can measure her ECG among other things. This morning Hulda gets more exhausted than normal, and this is detected by the system which automatically contacts the private healthcare group and puts Hulda in contact with the house doctor who can help her. Creating Dynamic Groups using Context-awareness 129

family

doctorHulda specialist

friends

Figure 10.1: A user-centric view of dynamic groups

10.1.2 Scenario 2 - Normal every-day-use

Anna is a teenage girl living at home with her parents. She is an active girl who likes to hang out with her friends, arrange parties and go horseback riding. This Tuesday, she meets up with a couple of friends downtown, where they discuss what to do on Friday evening. Anna proposes they should have a party at her house, though she has to check with her mum and dad ﬁrst. She picks up her dynamic groups enabled mobile phone and creates a protected group with the friends she’s meeting with. Later that night, after she checked with her parents, she sends a text-message to the group to let her friends know the party is on. On Thursday night Anna goes horseback riding. Once in the stable she decides to invite three of her stable friends to the party and invites them to the dynamic group. Anna notices a few more people have been invited to the dynamic group as the party-group is always up-to- date for all participants. After the party, and after all the funny photos and videos from the party have been shared with the group, the party-group fades away quickly (i.e. is not presented on top to the members) since the party is over and the group is no longer relevant. However, this party group has been added to Anna’s social network so she (or any of the other group members) can easily recreate the same dynamic group if she decides to have another party in the future.

10.1.3 Outline of the document

The rest of the paper is organized as follows: Section 10.2 introduces the dynamic group model and explains the concept. In Section 10.3 some implementation issues are discussed. Section 10.4 covers privacy and security issues. In Section 10.5 a short user evaluation is presented. The discussion is covered in Section 10.6 while conclusions and future work is covered in Section 10.7. 130 Creating Dynamic Groups using Context-awareness

10.2 The Dynamic Groups Model

There are currently several tools available for group communication, cooperation, and organization. Often these tools require users to know details about the contact they wish to reach. There are a number of ways in which people choose to ﬁnd the people they want to communicate with:

• The room metaphor: The classic room metaphor is where the user simply joins a room focusing on the topic of interest. The tools range from advanced tools such as group conferencing software [81, 107, 139] which allows for several media (e.g. video, audio, chat, etc.), to live chat groups [65] and even simple web-forums [113].

• The friend metaphor: Many of the popular friend metaphor tools today provide a simple search function which lets the user search for friends by username or contact details. The user can then add contacts to their "friends list" and then communicate with them. The tools started out as simple instant messaging applications [3, 64], but some have evolved and now also provide more advanced functions such as the possibility to send audio and video [87, 135]. • The tree metaphor: Some tools exist mainly for mapping social networks and can be categorized as tree metaphor tools. While some of these tools provide communication functionality [90, 102] the main functionality is to provide the user with an overview of their social network and show how the user ﬁts in as a node in a larger tree structure. Relations to other people can often be several different things, such as family member, friend, or colleague. Some of these tools also provide advanced search capabilities which helps the user ﬁnd other people [24, 76].

The authors seek to define a new conceptual communication model which interweave the three traditional communication metaphors dynamically, in a user friendly way. The definition of the dynamic group model is as follows: "The dynamic group model defines group communication where the participants can be invited based on context and social networks in both physical and virtual settings." The model has many similarities to existing solutions and can be used in pretty much the same way. However, a dynamic group offers new functionality such as the possibility for automatic group formation based on predefined rules. Simply put, the dynamic group model is a combination of all three metaphors, namely the room metaphor, the friend metaphor, and the tree metaphor. The user can maintain a friends list in parallel to groups, groups can be created much like rooms, and the history of past groups can help build an understanding of relations between users.

10.2.1 Forming and maintaining groups

Groups can be formed in many different ways. Co-located people could form a group easily by using Bluetooth or Near Field Communication on the currently used device, or it could be done manually by simply adding people to a group. A user could also form a group by Creating Dynamic Groups using Context-awareness 131 organizing contacts and making groups to simplify tasks for him-/herself. Finally, a group can be formed which will be open for other users to join, such a group could be a discussion group, or simply an interest group focusing on a speciﬁc topic. Since groups can be formed in a variety of ways the authors have deﬁned three main types of groups, namely Private, Protected, and Public:

• Private groups - the main purpose for a private group is to help organize contacts for a user. The private group is not visible to anyone else, only to the user. Group members of a private group do not know they are in this private group, nor do they know of each other. A simple use for a private group could be to form a temporary group to which a user could send vacation photographs or small messages. • Protected groups - the main purpose for protected groups is to make it easy to create secure groups between people who know of each other. All participants in the group can be seen as administrators and are able to invite other people, however the group is not searchable and it is not possible to join the group apart from being present when the group is formed or by getting an invitation from a member of the group. Protected groups are useful when friends meet and decide to organize an event later on, or for co-workers who decide to follow up on a meeting. • Public groups - the main purpose for public groups is to provide an open forum where things can be discussed. The creator of the group automatically gets the administrative role, but can assign administrative powers to other group members. A user can search for public groups based on keywords which can be the topic of the group, and join them. An example of a public group could be a public blog, or a group discussing a speciﬁc topic together.

Dynamic groups rely on an advanced user profile system where specific information about the user is listed. Such information can be user id, user name, other contact details, interests, expertise, and current location. This stored data is similar to what many currently available tools use, with the main difference being that this data is used to organize the contact list. To talk to a friend, or a number of friends, the user can simply select the contact(s) from a list, or search the contact list for specific criteria (e.g. friends who are interested in cars). To find a new contact the user can simply search the profile database for the desired criteria (e.g. people with interest in cars, or experts in computer science). However, there is no need to perform a query if the users are co-located as mentioned earlier in this section. Once a group becomes inactive it fades out from the user interface, but the information about the group is not removed unless the user explicitly removes it. Groups which have faded away can be recalled by searching or browsing for it. For private groups only the creator can recall it, for protected groups only those once part of it can recall it, and for public groups which have not explicitly been removed anyone can search for it. This history function helps maintain groups, as well as social networks. 132 Creating Dynamic Groups using Context-awareness

10.3 Implementation

Dynamic groups require several technical solutions to work properly. This section discusses the most essential parts required together with related work for each part. For storing contexts the authors propose to use HomeML[96], an Extensible Markup Language (XML) standard for sharing data between smart environments. For handling rules to provide automaticity the users propose to use HomeRuleML[51], a rule engine which has been integrated with HomeML. Finally, to handle the proﬁle system as well as query requests and results, the authors propose to develop HomeCom which will be integrated with HomeML as well.

10.3.1 Context-platform

The purpose of the context-platform is to store contexts from sensors in the environment. Some of the data, such as location data, is used in certain queries such as "start a dynamic group with the people close to me". Other data is needed to make it possible to make automatic decisions regarding the forming of dynamic groups. The scenario in Section 10.1.1 uses several contexts for its operation, such as the activity context "drinking tea", the time context "morning", and the location context "sitting on living-room sofa" to make the decision to create/join a dynamic "breakfast" group. Later on in the same scenario there is an example of how healthcare data is used for contacting healthcare professionals. The context-platform used for dynamic groups needs to be flexible and easy to configure and use in several different terminals. XML [40] offers such flexibility as it is an acknowl- edged standard for sharing of data over the Internet. Using XML as a front-end for sharing data between context-server and clients hides the complexity of the back-end implementation. Context Fabric (confab) [60] is one of the context-platforms which uses XML for the handling of data. Confab also includes an elaborate solution for handling of privacy issues, which unfortunately also makes the protocol quite complex and difficult to configure and update. Apart from Confab there are several other available context-platforms. These include Context Toolkit [34], Aware Home [70], Gaia [120], Aura [137], etc. These platforms were developed specifically for smart environments to provide context-aware services. However, they do not provide the flexibility that XML offers and make less suitable candidates for dynamic groups. For dynamic groups the authors propose the use of HomeML [96] an XML standard for sharing data between smart environments, see figure 10.2. This is an open standard for which researchers can propose changes and develop tools. One of the benefits of using HomeML for dynamic groups is that the standard is expected to be deployed in many smart environments around the world. Hence, this should simplify the integration of dynamic groups in the existing smart-environments. Additionally, several useful tools are being developed for this standard, which can be used to improve the functionality of dynamic groups. Creating Dynamic Groups using Context-awareness 133

SmartHome homeID + Must exist at least once + * Can exist any number of times Room roomID ? Optional * Nodes without any sign must exist once Device deviceID ? Description ? DeviceLocation

XPos YPos ZPos

DeviceType

Units

* RealTimeInformation runID

SampleRate StartTime EndTime

+ Event eventID

TimeStamp Data

Figure 10.2: Excerpt of the HomeML deﬁnition

10.3.2 Rule-engine

In a ﬁxed smart-environment it is often possible to pre-develop rules to function in the environment. This is true especially if the developers have knowledge of, or can control, which inhabitants will use the environment, their behaviour, etc. Dynamic groups however are meant to be open and dynamic, with the possibility of being customized to the individual who will use it. This means that the rules must be easy to create and customize for each and every individual, and it is also desirable to make editing rules so simple that the users can do this themselves. An interesting use of a rule engine is to add the possibility to have invitations show up on the terminal a user is active on, and to determine which communication media to use (e.g. video, sound, chat, etc.). The algorithm presented in "Supporting Automatic Media Resource Selection Using context-awareness" [72] can be used to achieve this. Many smart environments use rule-based systems for their logic handling [120, 137] but these are not very suitable for a general solution such as dynamic groups. Instead we turn to HomeRuleML [51], an open XML model (Figure 10.3 for representing rules which are meant to function together with HomeML. Since this is an open standard it is likely several tools will be developed which will help users develop advanced rules in a simple manner. Two graphical front-end concept tools, HomeCI [93] and HomeTL [122], for HomeRuleML already exist and further development of these might prove to be very useful for developing rules to dynamic groups. 134 Creating Dynamic Groups using Context-awareness

Rule ruleID + Must exist at least once * Can exist any number of times ? Optional Ref: Reference to another node Description Nodes without any sign must exist once DateRecordCreated

Ref: Operation

Outcome

? Value

? Action + Device deviceID

SetValue

(a) The HomeRuleML Rule-tree

Operation operator + Must exist at least once * Can exist any number of times ? Optional Ref: Reference to another node ? Time condition Nodes without any sign must exist once

Value

* Device deviceID

? Description

DeviceType

Units

+ Condition condition

Value

* Ref: Operation

* RuleReference ruleID

? Description

Condition condition

Value

(b) The HomeRuleML Operation-tree

Figure 10.3: The HomeRuleML deﬁnition Creating Dynamic Groups using Context-awareness 135

HomeRuleML uses sensor- and context-data which has been stored in a HomeML enabled smart environment. This means that HomeRuleML rules can focus on specifying the rules by deﬁning which sensors and contexts are of interest, instead of dealing with the data retrieval etc. The model is recursive which enables advanced logic expressions and the reuse of other, more generic, rules. Common logical operators (AND, OR, NAND, NOR, XOR, NOT, and a special operator ORDER) are being used to form the expressions, and common condition operators (greater than, less than, equals, not equals) are used to set calculate conditional expressions.

10.3.3 Proﬁle-system

Typically a profile system contains information such as address, telephone number, interests, etc. In most of these systems [64, 87, 135] it is not possible to query for users with specific information, the information serves no other purpose than to inform friends etc. Instead users add people to their friends list based on a user id. This works ok when adding people already known to a user, but not so well when trying to establish new contacts. What is needed in dynamic groups is a profile system which lets a user search for people based on several contexts and information. Further work has also been done in trying to find which people know what, etc. IKNOW [27] is a communityware tool which makes it possible for its users to search the network for answers to questions such as "who knows what?" and "who knows who knows what?". This is a powerful tool but also complex, which defeats the purpose of a simple profile system that can be easily queried for contacts. The authors propose a new model for representing user profiles called HomeCom. This model will be based on XML, similarly to HomeML and HomeRuleML which has previously been mentioned. This model will build on both the context-platform and the rule-engine to form a system in which a user can search for people based on the present information. The HomeCom model helps users manage groups and provide invite, add, remove, and query functionalities. It simplifies data handling to and from the HomeML server. Using XML for a profile system has several benefits. The hierarchical nature of XML makes it very suitable for representing user profiles. It also enforces a specific structure which in turn opens up for graphical user-interface developers to develop compatible clients for dynamic groups. There are of course some bandwidth concerns if there are several users querying for contacts using dynamic groups. One solution to this problem would be to use mirror servers. This could offer necessary backup, as well as load balancing. During initial tests the system can use one central server, and as dynamic groups are more accepted and used it can be extended with more servers. Of course these servers also need to incorporate necessary security solutions to prevent malicious users from altering the system and other users’ information.

10.3.4 Session management in clients

Integrating the proposed model with an existing communication tool or service can be done in various ways as long as the underlying system has support for group communication. To give 136 Creating Dynamic Groups using Context-awareness an example of how the dynamic group model can be used, this section brieﬂy describes how it can be integrated with a SIP-based communication system. To handle group communication many SIP-based systems use a centralized server running in the network to duplicate and route RTP streams to each other client. Brieﬂy put, implementing conferencing can be done in two different ways [39]:

• Dial-in - dial-in is based on letting each participating user send a SIP Invite message to the conference server in order to be part of the session. Users can also invite other users by sending a SIP Refer message asking them to join the conference. • Dial-out - dial-out lets the conference server invite all participating users by sending a SIP Invite message directly to them. In this case, to invite users to a session, the users typically sends a SIP Invite to the conference server containing a list of all other participating users.

In both the dial-in and dial-out case, some user needs to maintain a list of all participating users in order to invite them to the conference session. The proposed dynamic group model can thus be implemented by providing such a list to the clients. A list could for example be sent to the call control module which is a part of most clients.

10.3.5 Architectural overview

This section has described the essential parts of a dynamic groups architecture, as seen in Figure 10.4. Section 10.1.1 describes a smart environment which reports sensor data so the system can assist the user in every day life. This is combined with the ability to form dynamic groups with family, friends, and healthcare personnel. The sensors which are located in both user-carried mobile devices and the environment report data to the HomeML server. The HomeRuleML model listens to new data and checks the data according to the predefined rules. In the example when the user sits down in the living-room sofa the rule stored in Home- RuleML triggers. An XML query is sent to HomeCom containing a logical expression (e.g. "Breakfast AND Senior Citizens"). HomeCom parses the expression, searches the database, and returns available groups which match the query. By default, the groups are presented in order of activity with the most active group first, however the result can be sorted using other criteria. The rule is set to join the first group in the result list. In Section 10.1.2 an everyday scenario is described. Initially the user wants to form a group in order to plan a party. The user can form this group in a number of different ways, for instance by using built in Near Field Communication or Bluetooth, or even by selecting the members from a list. The user can also query for other people by sending a request to the HomeCom model. Creating Dynamic Groups using Context-awareness 137

Dynamic Group users

HomeCom

XML data (queries & result) L eM m Ho Sensor data (contexts) Sensor data

Rule data (new rules & result)

HomeRuleML

Figure 10.4: An overview of the parts which form the Dynamic Group system

10.4 Privacy and security

Privacy is an important factor when it comes to context-platforms, and profile systems. While there are solutions which consider this problem, such as Confab [60], the needs are slightly different in a dynamic groups architecture. Of course healthcare data needs to be unavailable to the public. It should also be possible for a user to choose which information to store in the context-platform. Pentland [109] has discussed ways of dealing with privacy issues in systems similar to dynamic groups. In order to deal with privacy concerns with information, two different solutions are proposed. Either all information is stored, and the user simply selects which information to make public afterwards, or the user can press a "mute"-button to prevent that information being made public. The authors argue that a user may choose to display different information for different groups, however, if a user chooses not to display certain information then they should not have access to similar information from others in the group. Security is also an important factor in dynamic groups. A user’s profile should be updated by the user only, and not by others. A user’s social network structure also needs to be secure from tampering as it contains information about user habits etc. Furthermore, some groups, such as public groups, may need to be moderated to make sure information is not spread to unwanted people and to minimize malicious behaviour. Furthermore it could also be possible for the creator of a group to define more fine grained access control where invitation rights etc. can be limited. Further research needs to be done in both security and privacy issues for dynamic groups. 138 Creating Dynamic Groups using Context-awareness

Figure 10.5: Screenshot of the prototype

10.5 User evaluation

To illustrate the functionality of dynamic groups the authors developed a simple prototype. The prototype consists of an interactive "paper-prototype" [141] which has the look and feel of a real prototype as seen in Figure 10.5, but only static content. The prototype shows how groups can be formed, how to add and remove contacts, which contexts a single user can provide, and also a map which can display the location of the users in a group, or a single user. Qualitative semi-structured interviews1 were held with 10 people (5 women and 5 men) between the ages of 21 and 53 (mean 34) with average or higher computer experience. The general opinion was that the concept would be quite usable though a few concerns like integrity and information overload were revealed. The results from the user evaluation has been divided into 4 different topics:

• Usage - While there was some initial scepticism amongst a few of the participants they did have some suggestions to what they would use the system for. The usage

1The results from the full interview can be found at http://media.csee.ltu.se/projects/mymates/ Creating Dynamic Groups using Context-awareness 139

ranged from leisure activities such as organizing events and meeting with friends, to security applications such as keeping contact with members within hunting parties, to professional applications such as organizing meetings etc. Some concerns were also raised regarding if the system would be wide spread enough upon deployment. In general the major benefit was thought to be the ability to contact a number of people at once. • Usability - The majority of the participants were quite satisfied with interface and usability aspects. However, one comment was raised regarding the content switching between screens, or more specifically that the layout and options changed. One person also commented on the quality of the graphics, and that these should be improved for the final product. • Functionality - Over the brief demonstrations there were few comments about additional functionality requirements or change of functionality. One comment was to add control over which media to be contacted on, such as choosing only to be contacted through chat messages from a specific group. Other comments had to do with interface design, and how to make active contacts and groups more easily accessible. The sug- gestion was to let inactive contacts and groups fade away (though accessible through scrolling down etc.). Finally, there was a desire to integrate the dynamic groups prototype with existing tools such as the contact lists and instant messaging applications. • Privacy - Several of the participants had privacy concerns with using the prototype. Many thought that the contexts being shared for public view could be misused by others. There was also some concern for sharing information in protected groups as an undesired person could get hold of a group member’s device and through that gain access to the information.

10.6 Discussion

Dynamic groups is a new concept developed by the authors. In this article the model has been described, and the major parts have been discussed. This section discusses the research questions presented in Section 10.1. How should dynamic groups be formed and maintained? Three different methods have been proposed for forming groups, private, protected, and public groups. Private groups are for a single user and are meant to help organize contacts, protected groups are mainly for co- located people (with the option of inviting additional people) which might want to maintain communication after they part, and public groups are for open groups which should be visible to, and searchable by, the general public. If a group is inactive for a while it will fade away, but users can maintain their groups by recalling groups from the history. How should sensor-data and contexts be stored to allow the functionality of dynamic groups? The authors argue that an open standard based on XML is the best choice for storing sensor-data and contexts for dynamic groups. Hence it has been decided to use HomeML which is developed for storing data from smart-environments, and the authors believe this 140 Creating Dynamic Groups using Context-awareness can be updated to encompass the information necessary for dynamic groups. Because it is an open standard the authors believe useful tools will be developed for handling HomeML data, furthermore using an open standard will help give dynamic groups the impact it needs. How should rules for automatic establishment of dynamic groups be created? Dynamic groups can be created based on user queries or based on previous constellations. A user can select an advanced query string which will be passed onto the database. The result will be given in form of an XML file which can be parsed and viewed with whatever user interface being used. Creating a dynamic group should be a simple process, and where there are predefined rules it can even be an automatic process. As for automatic generation of dynamic groups the authors propose to use HomeRuleML which is an open XML model for creating rules, it also works together with HomeML. What should a profile system contain to provide the functionality of the dynamic groups? The authors propose to develop a new model called HomeCom for handling user profiles. This model will be an open XML model similar to HomeML and HomeRuleML already being proposed for dynamic groups. In addition to user id, the model will also contain information such as contact-details, location, interest, expertise, etc. The user may simply choose not to share information which will present an unacceptable privacy risk to the user.

10.7 Conclusions and future work

The authors have presented a new model for group communication management called dynamic groups. This model provides advanced functionality for creating and maintaining a virtual social network, and also provides advances querying capabilities for connecting a user to desired people. Dynamic groups are intended for use in every-day-life, be it a healthcare environment, work, or spare-time activities. In addition an evaluation of the model has been performed on 10 people. The results from the evaluation has been positive, although a few concerns regarding privacy and information overflow were raised. The authors intend to further develop the dynamic group ideas, and refine the user requirements based on feedback from the evaluation. This includes implementing the HomeCom profile system, make the necessary functions for HomeRuleML and HomeML, and continue developing a working prototype for dynamic group systems. For the future prototype the authors also intend to incorporate leading solutions for information fusion in order to derive higher level contexts for use in HomeRuleML. Furthermore the authors wish to investigate if groups can be created autonomously by matching profiles and using ontologies. Finally the prototype will then be evaluated and the results, as well as developed schemas will be made available online to further the impact of dynamic groups.

10.8 Acknowledgements

This work has been partly supported through the MyMates Project and the Laboranova project. The authors would also like to thank Stefan Håkansson and Tommy Arngren at Ericsson Research in Luleå, as well as Arne Gylling and Johan Bengtsson at Centre for Creating Dynamic Groups using Context-awareness 141

Distance-Spanning Technology (CDT), for helpful input. Finally the authors would like to thank Zaheer ul Hussain Sani for the work on the prototype. 142 Part 11

Proﬁle Management for Dynamic Groups

Citation: Josef Hallberg, Mia Backlund-Norberg, Kåre Synnes, Chris Nugent, "Proﬁle Management in Dynamic Groups", Conditionally accepted to the special Springer volume on Intelligent Patient Management, 2009.

Contributions: I am the main author of this article. I created the HomeCom model and the privacy handling. The article was written in collaboration with Mia Backlund-Norberg, Kåre Synnes and Chris Nugent.

143

Proﬁle Management for Dynamic Groups 145

Proﬁle Management for Dynamic Groups

Josef Hallberg1, Mia Backlund-Norbergg1, Kåre Synnes1, Chris Nugent2 1 Luleå University of Technology, Sweden {josef.hallberg, mia.backlund-norberg, kare.synnes}@ltu.se 2 School of Computing and Mathematics, University of Ulster, N. Ireland [email protected]

Abstract

There is a growing need in supporting social networking for groups who often become isolated, such as elders living at home. In a social network people with similar diseases and ailments can find each other and share information to improve their understanding of their illness. Group communication tools can help maintain a virtual social network and provide a base for information retrieval. Nevertheless, they often lack the strengths of the social networking tools and vice versa. Within this work we have developed a new concept called dynamic groups. Dynamic groups make creation, management, and usage of groups for communication and social networking easy. Nevetheless, for this to work the profile management system is required to handle more than just user information, it is required to provide users with control over their data and offer privacy and customisation capabilities. This article presents HomeCom, a model for profile management in dynamic groups. It also presents the solutions for making queries, as well as the solutions for privacy and customisation using multiple profiles and an integrated rule engine.

11.1 Introduction

As we move towards a model of decentralised healthcare, where patients may prefer to remain at home rather than to move to a hospital or a nursing home, there is a greater need for mediated group communication. Mediated group communication supports remote healthcare and virtual social networks for patients by providing communication through a number of different media (e.g. audio, video, chat, etc.). Hence, mediated group communication is an important aspects for patient management. Healthcare personnel need to maintain contact with patients in their home and might be required to assess the severity of a situation through mediated communication. Further- more, studies have shown that loneliness can have a negative impact on a person’s wellbeing [78, 52] which is why supporting a virtual social network is important since many (e.g. elders and people with chronic diseases) feel isolated and may not be able to go outside because of for example their health or extreme weather conditions. Virtual social networks also provide an excellent base for information sharing, where patients can meet and discuss their illnesses 146 Profile Management for Dynamic Groups and ultimately improve the understanding of their illness, something they might not be com- fortable doing with a doctor present. There are currently a number of communication tools such as instant messaging tools (e.g. MSN [87], AIM [3], etc.) and web-tools (e.g. facebook [41] etc.) which can help us communicate with others through different media. Nevertheless, there are no tools which combine the strengths of simple communication tools with social networking tools and flexible group management. To achieve this, the authors have proposed dynamic groups, a model for flexible group communication which uses eXtensible Markup Language (XML) [40] for easy cross-platform implementation and simple third party development. The dynamic group model was first introduced in the article "Creating Dynamic Groups using Context-awareness" [50] and is defined as "group communication where the participants can be invited based on context and social networks in both physical and virtual settings". This model implies that virtual group communication can happen anywhere, either in a physical setting where people meet face-to-face, or in a virtual setting. Either way, group management should be quick, simple and flexible, even if the group is only temporary or if it is intended for longer use. Although dynamic groups can be used beyond the realms of healthcare, the communication model is well suited for healthcare applications as it can be used in conjunction with smart environments and use predefined rules and input from a sensorised environment to automatically set up group communication. Previous articles already present some of the necessary components, such as the context storage model, HomeML [96] , and the rule-engine, HomeRuleML [51]. This article focuses on profile management and privacy solutions, other important aspects of dynamic groups which are necessary for successful deployment. The rest of this article is organised as follows: Section 11.2 gives an overview of dynamic groups, Section 11.3 describes the profile management system, HomeCom, Section 11.4 gives an evaluation, and Section 11.5 presents some related work. Finally the discussion is presented in Section 11.6, and conclusions and future work are discussed in Section 11.7.

11.2 Background

This section describes previous work related to the concept of dynamic groups. It explains what dynamic groups are and why they are useful. It also describes the architecture and the necessary components required for dynamic groups to work. Security and media distribution protocols have been omitted from this overview. For an overview of these topics refer to the article "Creating Dynamic Groups using Context-awareness" [50].

11.2.1 Dynamic groups

A dynamic group can be viewed as the concept of being able to form groups on the users’ terms. We communicate with other people daily, some we meet face to face, others we communicate with through virtual means. Sometimes we need to organise things as a group, or simply just talk with more than person at the same time. Being able to communicate and Profile Management for Dynamic Groups 147 access expert competence relating to illnesses may become even more important for future generations who grow old. Dynamic groups therefore help us to create groups at a whim for just that purpose. The following presents a short scenario with the aim to better describe what dynamic groups are: Eve is an elderly woman living alone at home. She likes to travel but can not do it much anymore, instead she often looks at her old pictures from locations around the world. She likes embroidery and cooking and gladly shares her patterns and recipes with others. Her son and daughter live with their own families quite far away and can not visit very often. If it wasn’t for dynamic groups Eve would be quite lonely, however, the technology helps her keep in touch with both family and friends, and also supports her leisure activities. Eve starts her day in the living room sofa with a cup of tea and a sandwich. The dynamic group system recognises this behaviour and connects to a breakfast group which shows up on the TV, according to the predefined rule. After breakfast she picks up her embroidery. The pattern she is using came as a recommendation from a discussion group on sewing and embroidery that Eve is in. Time passes quickly and it is already lunchtime. Eve prepares some food and sits down to eat. The TV by the dinner table lights up and her son and daughter who are also eating lunch join from their own homes. In a few days Eve is going on a senior’s trip so there is much to prepare. She has already packed her camera and is planning to take plenty of pictures. The carer, who visits Eve daily, helps her set up a group consisting of friends and family to which she can send her pictures throughout her trip. The carer has also made sure to get Eve invited into a discussion group on age-related illnesses where Eve can find out more about her conditions. This afternoon when Eve is having a discussion a particular question is raised which none of the participants can answer, so Eve invites a geriatrician to the group in order to get the question answered. The discussion ends but Eve wishes to continue conversing. Since Eve has made good friends in this discussion group she decides to search for anyone in the group who might share her interest in cooking or embroidery. The scenario describes a number of different uses for dynamic groups:

• There is the private group which helped Eve organise contacts into a group which she can reuse over and over, for example by sending pictures throughout her trip. The members of a private group do not know of each other, it is only for the creator to help organise contacts. • There is the protected group which is a closed group for people who know each other. It is possible to join this group by being present when the group is created, or by being invited by someone who is already a member. Eve used a protected group to communicate with her son and daughter, and also for discussing age-related illnesses. • Finally there is the public group which is open for anyone. It is possible for anyone to search for this group based on information added about the group. Typically the creator or a number of assigned moderators will have moderation abilities for the group to keep it organised. Eve used a public group to ﬁnd recommendations for embroidery patterns and cooking recipes. 148 Proﬁle Management for Dynamic Groups

In addition, Eve can search for other people based on a number of criteria, such as interest and competence, and invite them to a group. Eve used this functionality to find a person in the discussion group with an interest in cooking or embroidery. This is one of the strengths with dynamic groups, along with the flexible group management system. It may be difficult to picture an elderly woman using a computer to search for information, however, it is important to appreciate that today’s current generation will also grow old and may have different needs.

11.2.2 Architecture

The dynamic groups system consists of mainly three different parts. There is the context storage infrastructure named HomeML [96], the rule engine named HomeRuleML [51], and finally the profile management system which is specified in this article, and is referred to as HomeCom. In addition to these there are of course protocols for media distribution and security, but these are outside the scope of this article. What is of interest for this article is how HomeML, HomeRuleML, and HomeCom match and work together. HomeML was originally developed as a standard for storing and sharing data between smart environments. It provides an approach whereby information about sensors, such as data and sensor location (both in the environment and on the body in the case of wearable sensors) can be stored in a structured and extensible manner. It also stores general information about the inhabitants themselves. HomeML exists as an XML schema which leaves the backend implementation open, although there are already several tools developed for this such as a HomeML database and browser. The purpose of HomeML for dynamic groups is to store sensor information to make group management easier and to provide contexts such as user location etc. It also serves as an information repository to be used by decision support rules. HomeRuleML is a model for decision support rules, meant to provide a structured approach in which rules can be shared between smart environments and hence save time on development, testing, and deployment. It integrates with HomeML and uses the information stored therein for input to its rules. It is a recursive XML schema which means a user can specify advanced and logical expressions. The purpose of HomeRuleML in the context of the dynamic groups is for creating services which will help users manage groups, but also to help users join groups automatically. An example would be in elderly care where a rule might state that whenever the user is drinking their morning tea whilst seated on the sofa the TV should turn on and connect to a breakfast group to provide some social interaction. The final component, HomeCom, can be seen as an addition to HomeML. It provides more detailed information about a user, such as detailed contact information, interests, and competence. It also provides an option to keep several different profiles to be used in different groups for privacy reasons. Furthermore, it provides a structure for search queries for finding groups and other users, as well as a structure for the results. In summary, HomeML stores context information which is received from sensor data and user generated data. HomeCom provides an added information layer for HomeML, focusing on user and group information as well as a query protocol. HomeRuleML uses the information from HomeCom and HomeML for decision support to help manage groups and to Profile Management for Dynamic Groups 149 provide automatic services for users. The overview of this information flow can be seen in Figure 11.1.

Dynamic Group users

HomeCom

XML data (queries & result) L eM m Ho Sensor data (contexts) Sensor data

Rule data (new rules & result)

HomeRuleML

Figure 11.1: An overview of the components which forms the Dynamic Group system

11.3 HomeCom

The profile management system (referred to as HomeCom) handles information about users and groups, and also provides a basis for advanced search queries. The profiles contain not only the necessary information about users and groups for search queries, but also have a built in privacy filter where users can specify which information should be visible and accessible to others. HomeCom, is like other dynamic group components modelled as an XML schema to make cross-platform implementation and third party development simple. It is important to note that HomeCom is not intended as is for end-users. It is merely a model which enables third party developers to create useful and compatible interfaces which can then be personalised and tailored to specific needs. This is a desired effect, as the number of advanced functions can be limited from a user perspective in order to simplify the user interface, however, can be incorporated in predefined rules. There are a number of important problems which need to be addressed when creating a profile handling system. The system needs to contain all the necessary and desired information, it must support flexible group formation, and of course provide users with necessary security and privacy solutions. Though security is a very important aspect it is outside the scope of this article. This article focuses on addressing the following research questions: • Which information should be contained within the profile system? 150 Profile Management for Dynamic Groups

• How should search queries be formulated and interpreted? • How should privacy be handled within the proﬁle management system?

The following section presents HomeCom, a proﬁle system for dynamic groups. It describes which information is necessary to store within the system as well as the privacy approach being used. Furthermore, it describes the implementation in the form of the development of a number of XML schemas which work together.

11.3.1 Proﬁle information

The profile should contain all information that might be used in a search query, since the main purpose of the profile is to give quick facts about a user or a group. For a user that means there should be contact information, quick facts about the person such as interests and competence, and also some contexts a user might want to show like current location. Some contextual information can be derived from the context-storage infrastructure (HomeML) and be used in conjunction with HomeCom. A group profile should contain information related to the topic the group covers, as well as a more extensive description. It is also useful to publish which media resources (e.g. video, audio, chat, etc.) are used in the group so people who are searching for a group can find one that matches their preferred way of communication. Furthermore, the group profile should also contain a list of current participants to provide a better overview of the group and help search queries. Finally, contrary to the user profile a group profile needs to contain connection information (for user profiles a connection can be negotiated between two clients directly if a user is online, if not it can be handled as offline messages through the server). Note that group profiles are only for public groups since neither protected or private groups are accessible for outsiders (except by invitation in the protected group case).

11.3.2 Privacy

Privacy in HomeCom is accommodated for by letting a user specify several different profiles. The user can select which information to add, have different aliases for different profiles, and specify which profile should be visible in which group. The default profile, which is visible to everyone, may contain very little information for people who are worried about privacy. Nevertheless, if a user joins a group with another profile, that information becomes available for other users in the group. This means if a user searches for someone they are more likely to find people already in a group with the user, since the group profile may contain more information than the default profile. It is also possible for a user to define rules for when certain information should be displayed. Such rules could be "only display my location if the user who is searching is displaying location", or "display all information about me if the user is in the same city as me". The latter rule would mean that someone who is searching will be able to find more detailed information about people in the vicinity and is therefore more likely to form a group with that person rather than someone else who matches the query. With a little modification these Profile Management for Dynamic Groups 151 rules can be processed using HomeRuleML which is already an important part of dynamic groups. The benefit of using HomeRuleML is that this model is created with the purpose of sharing rules, which means the dynamic group community could share rules between them. All in all, the privacy model used within HomeCom is very much based on the user making an active choice of which information should be visible where. Though this may sound complicated there are a number of functions which could help simplify the process. For instance, a user could create a profile with all information and make it so it is not shown anywhere. Then whenever the user is making a new profile for a group the user could copy information from the populated profile to the new profile. Also, users could reuse certain profiles in different groups and create a layered privacy model for themselves where one profile with much information is used in protected groups, and a less detailed profile could be used in public groups.

11.3.3 Implementation

HomeCom consists of many smaller parts due to the fact that it has many tasks to perform. It is required to represent relevant information about users and groups, it also needs to support several proﬁles for each user in order to provide the privacy layer discussed in the previous section. Finally, it handles queries for users and groups, as well as the results. Each of these parts is modelled as an XML schema, each of which will be discussed in the following subsections. The key which has been used in the pictures showing the schema can be seen in Figure 11.2.

Key + Must exist at least once * Can exist any number of times ? Optional Ref: Reference to another node Nodes without any sign must exist once

Figure 11.2: The key used in the schema

The Proﬁle Tree

The Profile tree (refer to Figure 11.3a) provides the foundation for information about users. A user can have several profiles, each profile identified by a profileID. The nodes in the Profiletree are all optional, though some can exist more than once. The reason for making the nodes optional is so users can choose which information to show to others, information which is supposed to be private is simply omitted. The fact that a user can maintain several profiles in parallel means the user can reveal different information in different groups by simply selecting which profile to make visible in a group. Most of the nodes are selfexplanatory, however, a few nodes are worth describing in more detail. The Interests-, and Competence-nodes can exist more than once. The purpose of these are to list a number of interests and competence areas the user has. These are important nodes 152 Profile Management for Dynamic Groups for dynamic groups as many of the queries will be based on either interest or on competence rather than name or alias. Finally, the Position-node can either be a static value (the Location- node) for users who simply just wish to list hometown or country, but can also be a link to a location sensor (the Device-node, identified by the deviceID) which will be able to provide more accurate and updated location information.

Profile profileID User userID

? + FirstName UserProfiles

? LastName Ref:Profile

? * Title ShowInGroup groupID

? * Alias UsingRule ruleID

? Gender Groups groupID

? Birthdate (b). The User schema

? Homepage

? Address

? Email

? Phone

? Mobilephone Group groupID

? ? Workphone Name

* ? Interests Topic

* ? Competence Description

? * Position MediaResources

? * Device deviceID Ref: Profile

? ? Location ConnectionInfo

(a). The Profile schema (c). The Group schema

Figure 11.3: HomeCom schemas

The User Tree

The User tree (refer to Figure 11.3b) ties together a user’s different profiles along with poli- cies for each profile, through the node UserProfiles which of course can exist more than once. Along with the reference to the Profile tree there is a node called ShowInGroup which specifies (using the groupID attribute) in which groups this profile is active. A user could of Profile Management for Dynamic Groups 153 course specify not to use the profile in any group and simply have the profile as an information repository to copy from when creating new profiles. In the UserProfiles node there is also a node called UsingRule which lets the user specify rules that should be applied to the profile. The attribute ruleID links to a rule stored within HomeRuleML which in turn could specify requirements for viewing parts of the profile, such as only displaying location to people in the vicinity. How this will work, along with required modifications to the HomeRuleML model, is discussed in Section 11.3.3. Finally there is a node called Groups identified by groupID which specifies all the groups the user is part of. When a user searches for others the system will search for default profiles which are publicly available, as well as user profiles which are available through the groups the user is part of. Assuming that profiles used in groups are more detailed than the public profiles, the user is more likely to find a matching query amongst people the user already shares a group with.

The Group Tree

The Group-tree (refer Figure 11.3c) is for relevant information about groups, and each group is identiﬁed by a unique groupID. The nodes Name, Topic,andDescription contain just what the names imply. The node MediaResources is used for specifying which media (e.g. chat, video, audio, etc.) are used for communication within the group. The Proﬁle-reference is used for listing all the participants within the group. Finally, the ConnectionInfo node contains the needed information for actually accessing the group. This could be for instance a SIP-address, or a URL.

The Query Protocol

The query protocols actually consist of three parts. The first part is the Query tree which defines the structure of a query, the second part is the Result tree which defines the structure of a reply from a query, and the third and final part is the QueryOperation tree which helps define logical expressions in a query. Modelling a query as XML might be unorthodox because of the overhead and added complexity compared to a simple keyword search. Neverthelkess, the authors believe the results will be more precise if the user can specify exactly which fields in a profile or group the user is interested in. The results follow the same principle, as it returns the matching profiles and not just profileID. This helps the user select which matches are best suited by looking at additional information. Of course interfaces can be created so users do not need to see the XML structures of neither query nor result. The QueryOperation tree (refer to Figure 11.4a) is a recursive schema intended for creating logical expressions. It has an attribute operator which can be one of the following logical operators: AND, OR, NOT, NAND (neither), NOR, XOR (either one or the other, but not both). These operators are applied to the information in the reference nodes Profile and Group, which means that if a user wishes to find people interested in cars or motorcycles the user would let the operator be OR and then specify two profiles, one with Interests = cars, another with Interests = motorcycles. If the user wishes to find people within a certain group it is possible to specify contexts within the Group node. For instance, a user looking for peo- 154 Profile Management for Dynamic Groups ple interested in cars within a motorists group could specify Group→Topic = Motorists and Group→Profile→Interests = Cars. If the user wishes to specify more complex queries it is possible to add more QueryOperation trees.

Query userID

* QueryOperation operator Ref: QueryOperation

* ? Ref: QueryOperation Ref: Profile Result

* ? * Ref: Profile Ref: Group Ref: Profile

* ? * Ref: Group SortBy Ref: Group

(a). The QueryOperation schema (b). The Query schema (c). The Result schema

Figure 11.4: HomeCom schemas

The Query tree (refer to Figure 11.4b) is identified by the user’s userID. This is so the system can access the groups the user is part of and hence search through profiles within these groups. A Query is quite similar to a QueryOperation with one exception, the SortBy node. The SortBy node helps a user specify which attribute is most important in the query and in which order the results should be organised. The valid values for this node are Activity (most active user or group first), Location (closest distance first), Details (most detailed profile or group first), Size (for groups only, largest group first), Acquaintances (for profiles only, people on contact list, or already sharing a group with the user, first). Finally, the Result tree (refer to figure 11.4c) is the result of a query. It contains references to profiles and groups which match the query. The results are ordered according to the SortBy information in the Query, with the best match first.

Updates to HomeRuleML

HomeRuleML [51] was originally intended for creating decision support rules for use within smart environments. It is built on top of HomeML [96] and relies on device data (e.g. sensor data) and logical expressions. In order to use HomeRuleML within the Profile-tree for specifying special conditions in which certain data should be shown there needs to be a number of updates to allow HomeRuleML to accomodate both Profile-, and Group-schemas. The nodes which have been added to the schema have been coloured in grey (Figure 11.5a and 11.5b). Figure 11.5a shows an excerpt of the Rule-tree used within HomeRuleML. This tree includes information about the rule, logical expressions, and an Outcome. Within the Outcome- node a new node has been added: the Profile-reference node. This means that if the rule is true the information in the Profile node is given to the recipient (the user who queried the system). Note that the profile given within the Outcome-node of the Rule-tree is not the same profile as the one listed in UserProfiles in the User-tree (Figure 11.3b). Instead, new information can be given in this profile (e.g. the Location could be specified in greater detail, etc.) if desired. Profile Management for Dynamic Groups 155

Operation operator

* ProfileCondition condition

? Ref: Profile

? Ref: Group

Rule ruleID * Device deviceID

? Description Description DeviceType DateRecordCreated Units Ref: Operation + Condition condition Outcome Value ? Ref: Profile * Ref: Operation

(a). An excerpt of the HomeRuleML (b). An excerpt of the HomeRuleML Rule-tree, with modifications in gray Operation-tree, with modifications in gray

Figure 11.5: Excerpts of the HomeRuleML trees, modiﬁcation in grey

To accomodate the logical expressions with profiles and groups the Operation tree (refer to Figure 11.5b) used in HomeRuleML also needs to be updated. The Operation tree is quite similar to the QueryOperation tree and the operator attribute can have the same values. The addition to this schema is mainly the addition of a conditional statement based on profiles and groups. The attribute of the ProfileCondition node is condition which can have the following operators: GT (greater than), LT (less than), EQ (equals), GEQ (greater than or equal to), LEQ (less than or equal to), and NOT. The profile of the user who’s making the query is compared to the values stored in either the Profile-, or the Group-nodes using the condition operator. If a user only wants to display their position to people living in London they would simply specify condition = EQ and Profile→Position→Location = London. This rule would then result in true for anyone who specified London as their location. For locations the system also deals with granularity. Therefore, in the rule the location can be set to City instead of London, which would then include any user with an address or coordinate within the same vicinity. The different granularity levels for Location are Street, Area code, City, Country. 156 Profile Management for Dynamic Groups

11.4 Evaluation

The HomeCom design presented in this article is the first version of the XML based schema. To facilitate the uptake of this model, the XML schema along with documentation providing guidance on use and steps to follow to make recommendations for changes to the current version, will be made available through a series of web based resources (http://trail.ulster.ac.uk/HomeML/). The authors also intend to make available any additional services, for example rules which have been developed for HomeCom. In addition to the HomeCom schema there are also a number of other tools available such as the HomeML schema, the HomeRuleML schema, and tools related to these. These include a number of assistive tools such as a browser and data editor which will help display and edit/populate HomeML data, a SQL database for efficient data storage and access on site, and tools for generating the HomeML files from the database. In addition a graphical interface for creating rules [93], will also be made available.

Eve Talbot E.T Female Embroidery Cooking Embroidery London

Figure 11.6: An example of a typical user proﬁle

11.4.1 Example scenarios

This section illustrates how the different parts of HomeCom can be used together. In the example given in Section 11.2.1 we have Eve, an elderly woman living at home. A typical profile for a woman like Eve could look like that as shown in Figure 11.6. In this profile Eve has chosen to leave certain information out, such as contact information, etc. Eve wishes this profile to be visible in two different groups, and there is also a rule which is to be applied. Eve is part of a total of three different groups. Profile Management for Dynamic Groups 157

Give more accurate location to people in the same city 2008-01-01 City ...

Figure 11.7: An example of a rule for HomeCom

The rule which Eve uses in her proﬁle relates to the granularity of her location. She does not want just anyone to know where she is, but she does not mind if people living in the same city know more details. The rule she is using (see Figure 11.7) allows her to give out her exact position recorded by her GPS device to people in the same city as herself. Note that she could set the granularity to something else, such as Street. One of the groups Eve is part of is the discussion group for age-related illnesses. This group is for elders living in the same area. Experts, such as geriatricians, can be invited to the group in order to answer questions. An example of how such a group may look like can be seen in Figure 11.8.

Geriatrics Age-related illnesses Discussion group for elders living in London ... ...

Figure 11.8: An example of a group

Finally, because Eve has become quite good friends with the others in the Geriatrics discussion group she decides to search for anyone who might share her interest in cooking or embroidery, and she wants to sort the results by Activity. Regardless of which user-interface 158 Proﬁle Management for Dynamic Groups

Eve uses, the end result of the query would look something like the code depicted in Figure 11.9. Note that since the system uses lazy evaluation, the search engine only needs to search the contacts within the Geriatrics group.

Cooking Embroidery Activity

Figure 11.9: An example of a query

In this manner Eve will be able to use dynamic groups in her everyday life. She can search for other people based on a number of contexts, she can be found on her terms (meaning she can have different proﬁles with different granularity and details). All in all, HomeCom is more than just a collection of information, it helps users manage and maintain dynamic groups. This means Eve will be able to create, manage, and join groups covering topics which are relevant to her, such as discussion groups on age-related illnesses.

11.4.2 Prototype

A prototype of the dynamic group application has been developed for the Nokia 6131 NFC edition phone. The prototype uses an early (unfinished) version of HomeCom for information storage. In addition, with the prototpye it is possible to add people to a group, join groups (both by invitation and by NFC) and communicate with individuals or groups through chat. In the version of HomeCom running in the prototype it is not possible to add rules to profiles, nor is the query capability which has been described in this article been made available. The screenshot seen in Figure 11.10 shows a simple interface where the user can choose to view the contact list, to view the groups, and to search. It also shows incoming messages and pending notifications.

11.5 Related work

This section presents work which is related to dynamic groups, and HomeCom. This includes group communication tools, webstandards and languages, and privacy solutions for proﬁle management. Proﬁle Management for Dynamic Groups 159

Figure 11.10: A screenshot of the prototype

11.5.1 XML, OWL, SOUPA

Like other dynamic group components HomeCom, is modelled as an XML schema. This is in order to simplify crossplatform implementation and make it possible for third party development. Nevertheless, XML is not the only web-language for this type of application. One alternative is the Web Ontology Language (OWL) [104] which is based on XML but with a few differences. OWL is designed to be used by applications that need to process the content of information. OWL facilitates greater machine interpretability of Web content than that supported by for example XML. It does this by providing additional vocabulary along with formal seman- tics. Chen et al [20] describe SOUPA, a shared ontology for supporting pervasive computing applications based on OWL. Although the idea behind SOUPA is good, it is very general, which in turn adds to complexity for some applications and tools, such as HomeCom. It is also using OWL which makes integration with existing XML based models somewhat difﬁ- cult. 160 Proﬁle Management for Dynamic Groups

11.5.2 Instant Messaging

Today there are a number of different communication tools. Instant messaging tools, like MSN [87] and AIM [3], supports synchronous communication (typically chat) to one or more individuals at the same time. Instant messaging tools typically use a contactlist as basis for who can send and receive messages to and from the user. The contact list also displays the current status of contacts (e.g. available, away, do not disturb, etc.). In social networking tools, for example facebook [41], the communication is typically asynchronous. In facebook there is a list of friends and there is a possibility to see who is online, but messages are either sent privately (more like an e-mail) or posted on a public wall (like a billboard), and not in synchronous chat-form. Furthermore, though there is a possibility to create groups, these are mainly for indicating belonging, and not for communication. Dynamic groups are different than both instant messaging tools and social networking tools in that they combine the strengths from both, though they are mainly intended for group communication. They offer the possibility to form and maintain groups, both for the purpose of indicating belonging and also for communication. The advanced proﬁle system also lets users deﬁne different information for different groups. Communication can be opened through a number of media (e.g. chat, audio, video, calendar events, etc.) and can be established either with an entire group, or individuals.

11.5.3 Privacy

When people fill out information about themselves to use a certain contact service, many do not consider that the information they provided to make it possible for friends to find them is also visible to others, some of which should not have access to the information. There is a difference between calling someone a friend online and a friend in the real world. As stated in [14] "Some people are willing to indicate anyone as Friends, and others stick to a conservative definition, most users tend to list anyone who they know and do not actively dislike. This often means that people are indicated as Friends even though the user does not particularly know or trust the person." For example, a facebook user made a script that asked 250 000 other users to add him as their friend, 75 000 users accepted [69, 46]. There are a number of different approaches to solve the privacy issue in profile management. With Identity Management a user’s profiles and identities are separated from services that are using them. The system allows people to define different identities and roles, associate personal data to it, and decide who to give the data to and when to act anonymously [152]. In [58] a policybased control of the users profile is used to maintain privacy. If access is denied to a certain service, due to lack of information about the user, the user is consulted on what actions to take. Marikawa et al. [86] suggest that privacy control is best achieved by using a combination of automatic policy-based access control and manual-based access control by the users. In HomeCom a user can create several profiles and assign these to different groups. The information in these profiles is only available to others in the same group. Furthermore, a user can define rules in which information can vary depending on the contexts of the user Profile Management for Dynamic Groups 161 who is watching the profile. All this is consolidated into a flexible privacy system where the user has control over his or her information and can define who gets to see what information.

11.6 Discussion

The current generation of young people have incredible access to information through the Internet. Computers are a big part of every day life, and people keep contact through instant messaging tools and other virtual communication means. When these people grow old it is not unlikely that they will keep using technology for communication and accessing information, for instance to find out more about their illness. Dynamic groups can be viewed as a way to make this possible, and with the possibility of deploying predefined rules and of developing a user interface on top of the models presented in this article, it makes it usable for our elderly persons, and those suffering form chronic diseases, today. In order to realise the concept of dynamic groups the authors have developed a number of models which specify data structures. HomeCom is another addition to these models, which specifies the structure for profile management within dynamic groups. Nevertheless, it is more than just a list of contexts about users and groups. Apart from the normal contact information etc. the information to be contained within the profile system also consists of a number of contexts such as Interests, Competence,andLocation. This information is available in order to utilise the power of dynamic groups where others sharing the same interest or experts in a certain area are never further away than a group invite. The Query protocol enables forming logical expressions to query for others. These logical expressions help limit the matches to those which are of interest, and to further help the user the matches can be sorted by Activity, Location, Details, Size,andAcquaintances. A user can also combine a query for a specific group and a query for a certain profile to limit the search further, for example to limit the search to a certain group. A query will search for default profiles (which may have limited information and details) as well as profiles which exist in groups where the user is a member. Assuming that profiles used in groups are more detailed than default profiles a user is more likely to find a relevant match amongst people already sharing a group with the user. Privacy is always a delicate matter when personal information is involved. In HomeCom this is handled by giving users control over their information and helping users set up different profiles which can be used in different groups. Having several profiles creates a layered privacy model where more detailed profiles can be used in groups where the user feels safe, and lesser detailed profiles can be used in more public places. With the possibility to add rules to profiles the user can also decide to display certain information to users who meet a certain criteria, hence creating a powerful customisation option for advanced users.

11.7 Conclusions and Future Work

This article has presented HomeCom, a profile management system for dynamic groups. It is implemented as a number of XML schemas with several parts for user profile informa- 162 Profile Management for Dynamic Groups tion, group information, and queries. Furthermore, it integrates with both HomeML, a model for context-storage, and HomeRuleML, a decision support model which also handles access management in user profiles. By providing HomeCom and the other parts of the dynamic group concept as open XML schemas it opens up for third party development of userinter- faces and backend systems, and encourages wider deployment and collaboration. To continue the development of dynamic groups and HomeCom the authors would like to update the current prototype with the new HomeCom capabilities such as rule handling and query capabilities. The authors would also like to make a user study with the updated prototype, where user comments help make the dynamic group concept even better. Furthermore, the authors would like to start collecting tools, services, and rules for HomeCom which can be distributed together with existing tools from http://trail.ulster.ac.uk/HomeML/ Part 12

Discussion, Conclusions and Future Work

163

Discussion, Conclusions and Future Work 165

12.1 Discussion

This thesis has presented pervasive computing technology and services for mediated communication in smart environments. The work has primarily been evaluated from a healthcare perspective. The following section provides a discussion on the research questions as presented in section 2.3.

1. What are suitable models for sharing of context and rules? A model is suitable for sharing if it utilises an accepted modelling method, if it is open for both use and modification (extension), and if it has a good impact (i.e. being widely used and accepted). Six different context modeling schemes were presented in section 2.2.1. Data representation in graphical models is not suitable for sharing because of its graphical nature, which would infer unnecessary overheads for parsing of the information. Logic-based models have complex data representations, which also make sharing complex and this would contradict the overarching design goal of reducing complexity in the system. Object-oriented models tend to be interwoven with the framework they are aimed for and are thus not suitable for sharing of data unless the same framework is used. Therefore, out of the models presented only three can be said to be suitable for data-sharing over the Internet: key-value models, markup-scheme models and ontology-based models. When comparing the remaining three models, key-value models do not offer hierarchical data structures, which makes them less suitable for sharing data collected from smart environments as such data is better organised into hierarchical levels (data from different rooms could be organized in a hierarchical structure for better overview). Ontology-based models offer great potential for data sharing, however, the purpose of these models is to classify and define relations between data. This adds unnecessary complexity to the context model which makes them less suitable for data sharing. This leaves markup scheme models as candidates for data sharing. Markup scheme models offer a readable text format, existing parsers, tools and support from a number of development platforms, which make them simple to use. Fur- thermore, they also offer a hierarchical structure, which makes them very suitable for data sharing [131]. XML is one of the more prominent markup scheme models available, and has therefore been used for the context- and rule-sharing models described in this thesis. The XML models presented in this thesis are intrinsically designed for sharing, so that they can be made publically available1 and open for development and modification to encourage usage. At the time of writing the models are being used in projects where the author is involved, such as the Cogknow project [25] and the MobiGroup project [85]. The Cog- know project targets people with mild dementia to help them navigate their day and utilises homeML to model context data. The MobiGroup project focuses on mobile group communication through usage of dynamic groups modeled by homeML, HomeRuleML and Home- Com for provisioning of decision support and profile management. The author believes homeML and HomeRuleML are suitable models for sharing context and rules because they are open, extendable, reusable, and provide a good representation of data. Furthermore, the models continue to improve as they are being used, and through the

1http://trail.ulster.ac.uk/HomeML/ 166 Discussion, Conclusions and Future Work use of the models in projects like Cogknow and MobiGroup the awareness of these models’ existence increases. This increases the impact of the models for context-sharing presented in this thesis.

2. How can we reduce complexity and cost of deploying technology and services into smart environments? A primary part of the cost and the complexity of deploying technology and services into smart environments are down to development and customisation of models. This is because smart environments utilises context models for context management, rule models used for reacting to changes in context, and services which are customised to the user needs. Customising a smart environment to a user might, however, be a great inconvenience as the user may have to leave their home for a period of time. The time spent on development and customising needs therefore be reduced, which will reduce complexity and cost of deploying technology and services into smart environments. Reducing the time spent on developing and customising models and services could be done by utilising knowledge sharing. Knowledge sharing has long been used for taking advantage of prior experience and to provide a knowledge-base from which new conclusions can be made. Knowledge sharing within smart environments would involve sharing of contexts, rules, and results from testing and data processing. Sharing such knowledge would contribute to the overall testing and evaluation of models, which ultimately improves the reliability of models and services in the smart environments. It also reduces complexity as sets of rules can be seen as a modules that are designed for reuse. The creation of rules may involve observing user behaviour, detecting common patterns, and analysing contextual data. Collecting enough context-data for this purpose can be time consuming. Nevertheless, a repository of contexts collected from a number of smart environments can reduce the time needed for data collection. However, in some cases there are people with knowledge in what rules are needed and which information is necessary for the rule (such as a carer who often cares for a patient). So, reducing the time required for creating or customising a rule can be done both by reusing rules and by using a tool for creating rules which can be used by the people possessing the knowledge of which rules are necessary. This thesis presents two models for context-sharing, namely HomeML which enables context-sharing between research organizations and HomeRuleML which enables sharing and reuse of rules. This thesis also presents HomeCI which is a graphical tool which helps the system experts (such as nurses) model rules using a non technical approach. The author believes that homeML, HomeRuleML, and HomeCI will decrease cost and complexity of deploying technology into smart environments if used. HomeML help provide a knowledge base which can reduce the time needed to collect data for analysing, observing user behaviour, and creating models. HomeRuleML offers a model for reusable rules which reduce development time, and HomeCI reduce the complexity of creating rules. Reduc- tion in development time and complexity for creating models and rules ultimately leads to reduced complexity and cost of deploying technology and services into smart environment. Discussion, Conclusions and Future Work 167

3. How can we overcome limitations of current handheld communication devices? Handheld communication devices are small and easy to carry around but their size is also cause for limitations, such as the small screen, limited input capabilities, and limited battery life. Furthermore, trying to capture quality video using the built-in camera on a handheld communication device might be difﬁcult. For people on the move who have a need for rich communication something better is required. One solution to overcome these limitations is to utilise other resources, such as those found in the environment, in conjunction with the handheld device. If a user desire higher quality video, then use the video stream from a camera located in the environment. And if the user needs a larger display for a picture, then show the picture on a monitor in the environment. A handheld communication device that utilise such media resources in the environment can offer a richer communication and thus overcome limitations such as the small screen or limited input capabilities. A smart environment is required to support a discovery mechanism so that available media resources and computational devices in the environment can be found and conﬁgured for dynamic use. Central to this discovery mechanism is an algorithm that selects the most appropriate resources to use in order to reduce the complexity for the user. This thesis present a solution where media resources in the environment are used to satisfy the user’s communication needs. The media resources are found using context-awareness, such as awareness of the user’s location and available media resources at the location. A media resource selection algorithm assist the user by proposing appropriate media resources to use based on the user’s preference on privacy, cost, and quality. The author claim that using media resources in the environment is a solution to overcome the limitations in handheld communication devices, assuming there are available media resources in the environment that match the user needs. Furthermore, the author also claims that to reduce the complexity of choosing which media resource to use, the system should assist the user by proposing appropriate media resources. This is addressed by the media resource selection algorithm which is presented in this thesis.

4. How can we improve communication within smart environments? Cooperation in a distributed environment offers some interesting challenges, such as providing remote information access and mediated group communication. Furthermore, when a physical meeting ends it might be desirable to continue collaboration in a virtual meeting, and when there is a need for further information it might be desirable to invite someone who pos- sesses such information. Conventional communication tools, such as the telephone, might not be sufficient for this purpose as the functionality for achieving rich communication is limited, and the traditional telephone service offers poor support for flexible group communication. This thesis presents a wearable computing solution which provides access to distributed information and mediated communication. This allows a user to move around without loos- ing these capabilities as the user is carrying all the necessary equipment, and only relies on the smart environment to provide context information. This thesis also presents a solution for using media resources in the environment, which offers similar capabilities but without requiring the user to carry any heavy equipment. By adapting new media resources as the user changes location the session can follow and continue to provide both information access and mediated communication. This solution relies on the smart environment to provide both 168 Discussion, Conclusions and Future Work context-information and media resources. For group communication needs this thesis presents a new concept of communication called dynamic groups. Dynamic groups offer flexible group communication where groups can be formed, and new users can be invited, based on users’ contexts and profile information. Dynamic groups also provide the functionality of a rule engine, in the form of HomeRuleML, which means that it is a tool that can be used by people with little or no computer experience as communications can be set up automatically according to predefined rules. This thesis suggests that there are clear benefits for communication and cooperation within smart environments when using mobile and wearable devices together with tools supporting media resources in the environment and dynamic groups. The benefits include better information access, richer communication, and less complex group communication, which all contribute to improve communication within smart environments.

5. How can we improve healthcare using smart environments? In times of cutting costs on healthcare, there is a need of optimising healthcare services while improving the quality of the care. Hence, making healthcare services more efﬁcient is desirable, and decreasing the load on our hospitals and care centres by preventing hospitalization and institutionalization may improve the quality of life of those affected. In this thesis the focus has been mainly on improving communications using smart environments, and providing remote healthcare monitoring. Improving the way we communicate with both information and between people, may decrease the time spent on information retrieval, which for example would leave nurses more time to spend with patients. Deploying technology and services into smart environments may help prolonging the time a patient can stay in their home and by doing such hopefully can improve their quality of life. Smart environments can provide information about the surrounding environment and the user in addition to supporting interaction with devices in the environment. Providing remote care monitoring may be an issue of having carefully placed sensors in the environment which can measure user behaviour and other user data. Smart clothing, where sensors are attached to the patient’s clothes, could measure different vital signs and movement related data about the user and relay this information to a central hub for analysis so behavioural and health analysis can be performed. The research presented in this thesis can help collect data about both vital signs and movements, analyze it, and issue warnings or recommendations when and where appropriate and through a number of different mediums. Dynamic groups even enable a communication session, using appropriate media resources, to be established automatically with a doctor in emergencies. By observing user behaviour and identifying daily activities, as presented in Part 9, the system could remind the user of the next step to take, or even intervene when something goes wrong. The system could for example warn when the stove is left turned on and the person is distracted by a ringing phone (a typical situation for dementia sufferers). The experience gained from deploying and evaluating smart environments for elderly and persons with dementia indicates that healthcare indeed can be improved. This thesis therefore concludes that remote healthcare monitoring and personalized services supported by a smart environment in the home or at a care organization is increasingly important to meet future Discussion, Conclusions and Future Work 169 requirements on the healthcare systems. To that end, such smart environments needs to be made less expensive and less complex to deploy, while supporting new technologies and services such as tools for rich and dynamic communication.

12.2 Conclusions

This thesis presents novel solutions for smart environments. It proposes sharing of contextual data and rule conﬁguration through open models to reduce complexity and cost for development and deployment of pervasive technology and services. It also proposes new concepts for mediated communication which utilises mobile devices together with devices in the smart environment to improve usability and functionality of communication services. The main scientiﬁc contributions of this thesis can be summarized as follows:

• A collection of models and tools that form the openHome Suite. The suite support context- and rule-sharing through homeML and HomeRuleML. It also include tools for database management such as a data browser and an analytical tool as well as a graphical front-end for creating of rules (HomeCI). • A new concept of using media resources in the environment to improve communication capabilities without adding cumbersome equipment that the user has to carry. This new concept includes an algorithm which can assist the user with selecting the appropriate resources in the environment based on user preferences. • A new concept for group communication called dynamic groups. This allows for groups to be formed both in physical and virtual settings, and enables simple and ﬂex- ible group communication. Furthermore, by utilising the strength of smart environments and rules (HomeRuleML) the system can be made automatic to further simplify usability of the system, and even provide safety.

This thesis suggests that sharing analytical models, rules and tools can improve the overall quality and reliability of models through thorough use and continuous improvements. Building a repository of shared knowledge and contexts can simplify deployment since the repository will provide a number of resources which can be shared and reused. Furthermore, this thesis suggests that mediated communication can be improved using smart environments, both through devices available in the environment and through context information and rules. Looking at a wider perspective and the possible impact that the proposed technologies can have on the society, the presented research can be seen as a step towards a wider deployment of smart environments and an increase of pervasive services to assist in undertaking activities of daily living. The ability to use resources in the environment for communication is greatly beneﬁcial for people in need of provision of rich communication capabilities but are unable to carry the necessary equipment. Dynamic groups have many uses, both professionally and for leisure activities. Nevertheless, the ability to combine group communication with automatic functionality makes it especially useful for environments where the user might be unable to operate the technology, either due to lack of knowledge or from simply being unable to use the controls. 170 Discussion, Conclusions and Future Work

There will always be a requirement to increase the speed of development and deployment of models, rules, and services. Furthermore, the desire to communicate is not expected to rad- ically change in the future. Therefore, the models for context-sharing, and the new concepts for mediated communication are applicable to present and future technology alike, where a smart environment exists to provide the necessary functionality. By this, I conclude this thesis by stating that smart environments hold great potential for the future, and the research presented herein provides important components for its success.

12.3 Future Research Directions

The models for context-sharing developed within the scope of this thesis have been deployed in various research projects. Nevertheless, it is important to continue to deploy these in as wide a setting as is possible. Setting up a number of smart environments which start to share data, analyze data, and build a repository of models, rules, and tools is essential to achieve any level of impact. This would also lead to new levels of needs being uncovered and subsequently to new research to address these. Furthermore, this would also test the validity of the models, and test the performance of the tools which have already been developed. Combining the media resource discovery algorithm with dynamic groups and smart environments is an interesting prospect. Being able to establish a group communication session automatically with the most suitable media resources in the environment creates a very flexible system which would enable a user to move around while the communication session is following by migrating between media resources. While this may be very useful in a home with only one inhabitant, it might cause a privacy problem in a home or other environment which may contain several people. Nevertheless, many elders live at home alone and could benefit from a communication system which would always ensure that the most suitable media resources were being used, both for convenience and for safety purposes. I would like to see both the models for context-sharing and the new concepts for mediated communication deployed in a wider scale to achieve the impact they were intended to have. In order for this to take place further development, testing, and evaluation is necessary since the work presented in this thesis is mostly at a prototype stage. Nevertheless, even if the research is not deployed in a wide scale it is published and will serve as inspiration for similar research. At present the area of data sharing and interoperability is key within the realms of Connected Health [26] and it is hoped that this work will offer some insight into this area. It is the belief of the author that context-sharing in some form will be necessary for the future of smart environments, and that our mobile devices will not suffice in some situations and mediated communication using smart environments will be necessary. Bibliography

171

Bibliography

[1] G. D. Abowd, A. K. Dey, P. J. Brown, N. Davies, M. Smith, and P. Steggles. Towards a better understanding of context and context-awareness. In HUC ’99: Proceedings of the 1st international symposium on Handheld and Ubiquitous Computing, pages 304–307, London, UK, 1999. Springer-Verlag. [2] Ageing and life course, world health organisation. http://www.who.int/ageing/en/, November 2008. [3] AIM. http://www.aim.com, November 2008. [4] J. C. Augusto and C. Nugent. Designing Smart Homes: The Role of Artificial Intelli- gence. Springer, 2006. [5] F. Axisa, P. Schmitt, C. Gehin, G. Delhomme, E. McAdams, and A. Dittmar. Flexible technologies and smart clothing for citizen medicine, home healthcare, and disease prevention. IEEE Transactions on Information Technology in Biomedicine, 9:325– 336, 2005. [6] J. Bauer. Identification and Modeling of Contexts for Different Information Scenarios in Air Traffic. PhD thesis, Technische Universität Berlin, 2003. [7] C. Becker, M. Handte, G. Schiele, and K. Rothermel. Pcom - a component system for pervasive computing. In PERCOM ’04: Proceedings of the Second IEEE In- ternational Conference on Pervasive Computing and Communications (PerCom’04), page 67, Washington, DC, USA, 2004. IEEE Computer Society. [8] C. Becker and D. Nicklas. Where do spatial context-models end and where do ontologies start? a proposal of a combined approach. In In Proceedings of the First International Workshop on Advanced Context Modelling, Reasoning and Management in conjunction with UbiComp 2004, 2004. [9] C. Becker and G. Schiele. Middleware and application adaptation requirements and their support in pervasive computing. In ICDCSW ’03: Proceedings of the 23rd In- ternational Conference on Distributed Computing Systems, page 98, Washington, DC, USA, 2003. IEEE Computer Society. [10] C. Becker, G. Schiele, H. Gubbels, and K. Rothermel. Base " a micro-broker-based middleware for pervasive computing. In PERCOM ’03: Proceedings of the First IEEE

173 174 Bibliography

International Conference on Pervasive Computing and Communications, page 443, Washington, DC, USA, 2003. IEEE Computer Society. [11] L. Belochi. Telemedicine Glossary: Glossary of Concepts, Technologies, Standards and Users. 5th ed. European Commission, 2003. [12] L. Black, C. McMeel, M. McTear, N. Black, R. Harper, and M. Lemon. Implementing autonomy in a diabetes management system. Journal of Telemedicine and Telecare, 11:6–8, 2005. [13] L. Blonde. Removing polytherapy as a barrier to adherence. Managed Care, Compli- ance and Persistence with Medication Therapy, 9:1, 2000. [14] D. M. Boyd. Friendster and publicly articulated social networking. In CHI ’04: CHI ’04 extended abstracts on Human factors in computing systems, pages 1279–1282, New York, NY, USA, 2004. ACM. [15] S. Brady, L. Dunne, A. Lynch, B. Smyth, and D. Diamond. Wearable sensors? what is there to sense? Personalised Health Management Systems: The Integration of Innovative Sensing, Textile, Information and Communication Technologies, pages 80– 88, 2005. [16] P. Budde. Global mobile communications - statistics, trends and forecasts. Communi- cation Pty Ltd, 2008. [17] F. Cabitza, M. Sarini, and B. D. Seno. Djess—a context-sharing middleware to deploy distributed inference systems in pervasive computing domains. In IEEE Pervasive Services (ICPS’05), pages 229–238, 2005. [18] H. Chen, T. Finin, and A. Joshi. An Ontology for Context-Aware Pervasive Computing Environments. In IJCAI 03 Workshop on Ontologies and Distributed Systems, 2003. [19] H. Chen, T. Finin, and A. Joshi. Using owl in a pervasive computing broker. In Workshop on Ontologies in Agent Systems, AAMAS-2003, 2003. [20] H. Chen, F. Perich, T. Finin, and A. Joshi. Soupa: Standard ontology for ubiquitous and pervasive applications. In In International Conference on Mobile and Ubiquitous Systems: Networking and Services, pages 258–267, 2004. [21] P. P.-S. Chen. The entity-relationship model—toward a uniﬁed view of data. ACM Trans. Database Syst., 1(1):9–36, 1976. [22] K. Cheverst, K. Mitchell, and N. Davies. Design of an object model for a context sensitive tourist guide, 1999. [23] E. Chtcherbina and M. Franz. Peer-to-peer coordination framework (p2pc): Enabler of mobile ad-hoc networking for medicine, business, and entertainment. In In Pro- ceedings of International Conference on Advances in Infrastructure for Electronic Business, Education, Science, Medicine, and Mobile Technologies on the Internet (SS- GRR2003w) (LŠAquila/Italy, 2003. Bibliography 175

[24] Classmates. http://www.classmates.com, November 2008. [25] Cogknow. http://www.cogknow.eu, November 2008. [26] Connected Health. http://www.connected-health.org/, November 2008. [27] N. Contractor, D. Zink, and M. Chan. IKNOW: A Tool to Assist and Study the Cre- ation, Maintenace, and Dissolution of Knowledge Networks. In Sunbelt Social Net- works Conference, 1998. [28] D. J. Cook and S. K. Das. How smart are our environments? an updated look at the state of the art. Pervasive Mob. Comput., 3(2):53–73, 2007. [29] D. Craig, C. Nugent, and M. Mulvenna. Healthcare technologies for older people: what do physicians think? In Proceedings of the 4th International Conference on Smart homes and health Telematics, ICOST2006, pages 331–334, 2006. [30] R. L. Daft and R. H. Lengel. Organizational information requirements, media richness and structural design. Manage. Sci., 32(5):554–571, 1986. [31] N. Dahlbäck, A. Jönsson, and L. Ahrenberg. Wizard of oz studies: why and how. In IUI ’93: Proceedings of the 1st international conference on Intelligent user interfaces, pages 193–200, New York, NY, USA, 1993. ACM Press. [32] C. Dewes, A. Wichmann, and A. Feldmann. An analysis of Internet chat systems. In IMC ’03: Proceedings of the 3rd ACM SIGCOMM conference on Internet measurement, pages 51–64, New York, NY, USA, 2003. ACM Press. [33] A. K. Dey. Understanding and using context. Personal and Ubiquitous Computing, 5(1):4–7, 2001. [34] A. K. Dey, D. Salber, and G. D.Abowd. A conceptual framework and a toolkit for supporting the rapid prototyping of context-aware applications. Anchor article of a special issue on context-aware computing in the Human-Computer Interaction (HCI) Journal, 16:97–166, 2001. [35] A. Dittmar, F. Axisa, G. Delhomme, and C. Gehin. New concepts and technologies in home care and ambulatory monitoring. Wearable eHealth Systems for Personalised Health Management: State of the Art and Future Challenges, pages 9–35, 2004. [36] M. Donnelly, C. Nugent, D. Finlay, P. McCullagh, and N. Black. Making smart shirts smarter: Optimal electrode placement for cardiac assessment. International Journal of Assistive Robotics and Mechatronics, 8:53–60, 2007. [37] M. Drugge, J. Hallberg, P. Parnes, and K. Synnes. Wearable Systems in Nursing Home Care: Prototyping Experience. IEEE Pervasive Computing, 5(1):86–91, Jan-Mar 2006. [38] D. Enström, A. Nohlgren, H. Olofsson, J. Peisa, and P. Synnergren. Multimedia telephony for IMS - Interoperable VoIP with multimedia support. In Ericsson Review No. 2, 2007. 176 Bibliography

[39] A. J. et al. Session Initiation Protocol (SIP) Call Control - Conferencing for User Agents (RFC 4579). In RFC 4579, 2006.

[40] Extensible markup language (xml). http://www.w3.org/XML/, November 2008. [41] Facebook. www.facebook.com, November 2008. [42] M. Feki, B. Abdulrazak, and M. Mokhtari. XML modelisation of smart home environment. In Proceedings of the 1st International Conference on Smart homes and health Telematics, pages 55–60, 2001. [43] H. R. Fischer, S. Reichlin, J.-P. Gutzwiller, A. Dyson, and C. Beglinger. Telemedicine as a new possibility to improve heatlh care delivery. M-Health: Emerging Mobile Health Systems, pages 203–218, 2006. [44] D. Garlan, D. P. Siewiorek, and P. Steenkiste. Project aura: Toward distraction-free pervasive computing. IEEE Pervasive Computing, 1:22–31, 2002.

[45] B. Grosof. Representing e-commerce rules via situated courteous logic programs in ruleml. Electronic Commerce Research and Applications (ECRA), pages 2–20, 2004. [46] R. Gross, A. Acquisti, and J. Heinz. Information revelation and privacy in online social networks. In WPES ’05: Proceedings of the 2005 ACM workshop on Privacy in the electronic society, pages 71–80, New York, NY, USA, 2005. ACM Press. [47] T. R. Gruber. Toward principles for the design of ontologies used for knowledge sharing. Int. J. Hum.-Comput. Stud., 43(5-6):907–928, 1995. [48] T. Gu, H. K. Pung, and D. Q. Zhang. A middleware for building context-aware mobile services. In In Proceedings of IEEE Vehicular Technology Conference (VTC, 2004. [49] T. Gu, X. H. Wang, H. K. Pung, and D. Q. Zhang. An ontology-based context model in intelligent environments. In In Proceedings of Communication Networks and Dis- tributed Systems Modeling and Simulation Conference, pages 270–275, 2004. [50] J. Hallberg, M. B. Norberg, J. Kristiansson, K. Synnes, and C. Nugent. Creating dynamic groups using context-awareness. In MUM ’07: Proceedings of the 6th international conference on Mobile and ubiquitous multimedia, pages 42–49, New York, NY, USA, 2007. ACM. [51] J. Hallberg, C. Nugent, R. Davies, K. Synnes, M. Donnelly, D. Finlay, and M. Mul- venna. HomeRuleML - A Model for the Exchange of Decision Support Rules Within Smart Environments. In 3rd annual IEEE Conference on Automation Science and Engineering (CASE), pages 513–520, 2007. [52] L. Hawkley and J. Cacioppo. Loneliness and pathways to disease. Journal of Brain, Behavior and Immunity, 17(17):98–105, 2003. Bibliography 177

[53] Health level seven. http://www.hl7.org, November 2008.

[54] S. Helal, W. Mann, H. El-Zabadani, J. King, Y. Kaddoura, and E. Jansen. The gator tech smart house: A programmable pervasive space. Computer, 38(3):50–60, 2005. [55] A. Held and E. Schill. Modeling of context information for pervasive computing applications. In Proceedings of 6th World Multiconference on Systemics, Cybernetics and Informatics (SCI2002), 2002.

[56] K. Henricksen, J. Indulska, and A. Rakotonirainy. Modeling context information in pervasive computing systems. In Pervasive ’02: Proceedings of the First International Conference on Pervasive Computing, pages 167–180, London, UK, 2002. Springer- Verlag. [57] K. Henricksen, J. Indulska, and A. Rakotonirainy. Generating context management infrastructure from high-level context models. In In 4th International Conference on Mobile Data Management (MDM) - Industrial Track, pages 1–6, 2003. [58] N. Henze and D. Krause. User proﬁling and privacy protection for a web service oriented semantic web. In In Proceedings of 14th Workshop on Adaptivity and User Modeling in Interactive Systems (ABIS), 2006. [59] C. K. Hess, M. Román, and R. H. Campbell. Building applications for ubiquitous computing environments. In Pervasive ’02: Proceedings of the First International Confer- ence on Pervasive Computing, pages 16–29, London, UK, 2002. Springer-Verlag. [60] J. I. Hong and J. A. Landay. An architecture for privacy-sensitive ubiquitous computing. In MobiSYS ’04: Proceedings of the 2nd international conference on Mobile systems, applications, and services, pages 177–189, New York, NY, USA, 2004. ACM Press. [61] X. Hong, C. Nugent, M. Mulvenna, S. McClean, B. Scotney, and S. Devlin. Assess- ment of the impact of sensor failure in the recognition of activities of daily living. Lecture Notes in Computer Science, 5120:136–144, 2008.

[62] X. Hong, C. Nugent, M. Mulvenna, S. McClean, B. Scotney, and S. Devlin. Evidential fusion of sensor data for activity recognition in smart homes. In Pervasive and Mobile Computing, 2008. [63] J. Hughes, V.King, T. Rodden, and H. Andersen. The role of ethnography in interactive systems design. interactions, 2(2):56–65, 1995.

[64] ICQ. http://www.icq.com, November 2008. [65] Internet Relay Chat Protocol. http://www.ietf.org/rfc/rfc1459.txt, November 2008. [66] S. Intille. Designing a home of the future. IEEE Pervasive Computing, 1:76–82, 2002. 178 Bibliography

[67] Jboss rules (drools 3.0). http://www.jboss.org/drools/docs/, November 2008. [68] Jsr 94: Java rule engine api. http://jcp.org/en/jsr/detail?id=94, November 2008. [69] K. Jump. A new kind of fame. Columbia Missourian, 2005. [70] C. D. Kidd, R. Orr, G. D. Abowd, C. G. Atkeson, I. A. Essa, B. MacIntyre, E. D. Mynatt, T. Starner, and W. Newstetter. The aware home: A living laboratory for ubiquitous computing research. In Proceedings of the Second International Workshop on Cooperative Buildings, Integrating Information, Organization, and Architecture, pages 191–198. Springer-Verlag, 1999. [71] Konnex - open standard for home and building control. http://www.konnex.org, November 2008. [72] J. Kristiansson, J. Hallberg, S. Svensson, K. Synes, and P. Parnes. Supporting Au- tomatic Media Resource Selection Using Context-Awareness. In In 3rd Interna- tional Conference on Advances in Mobile Multimedia (MoMM2005), pages 271–282, September 2005. [73] M. Langheinrich. Privacy by design - principles of privacy-aware ubiquitous systems. In UbiComp ’01: Proceedings of the 3rd International Conference on Ubiqui- tous Computing, pages 273–291, London, UK, 2001. Springer-Verlag. [74] J. K. Lee and M. M. Sohn. The extensible rule markup language. Commun. ACM, 46(5):59–64, 2003. [75] J. Li, K. Yu, T. He, Y. Lin, S. Li, and Y.-Q. Zhang. Scalable portrait video for mobile video conferencing. IEEE Transactions on Circuits and Systems for Video Technology, 13(5):376–384, May 2003. [76] LinkedIn. http://www.linkedin.com, November 2008. [77] J. Lowrance, T. Garvey, and T. Strat. A framework for evidential reasoning systems. In Proceedings of the 5th National Conference of the American Association for Artiﬁcial Intelligence, pages 896–903, 1986. [78] E. Mahon, A. Yarcheski, and J. Yarcheski. Health consequences of loneliness in ado- lescents. Journal of Research in nursing and health, 16(1):23–31, 1993. [79] P. Maniatis, M. Roussopoulos, E. Swierk, K. Lai, G. Appenzeller, X. Zhao, and M. Baker. The mobile people architecture. ACM Mobile Computing and Commu- nications Review, 3:36–42, July 1999. [80] Marratech. http://www.marratech.com, November 2008. [81] S. McCanne and V. Jacobson. vic : A ﬂexible framework for packet video. In ACM Multimedia, pages 511–522, 1995. Bibliography 179

[82] J. McCarthy and S. Buvac. Formalizing context (expanded notes). Technical report, Stanford University, Stanford, CA, USA, 1994. [83] S. McClean and B. Scotney. Using evidence theory for the integration of distributed databases. International Journal of Intelligent Systems, 12:763–776, 1997. [84] B. Meade and J. Dunbar. A virtual clinic: telemetric assessment and monitoring for rural and remote areas. The International Electronic Journal of Rural and Remote Health Research, Education, Practice and Policy, 2004. [85] MobiGroup. http://media.csee.ltu.se/projects/MobiGroup/, November 2008. [86] D. Morikawa, M. Honjo, A. Yamaguchi, and M. Ohashi. A proposal of user proﬁle management framework for context-aware service. In SAINT-W ’05: Proceedings of the 2005 Symposium on Applications and the Internet Workshops, pages 184–187, Washington, DC, USA, 2005. IEEE Computer Society. [87] MSN. http://www.msn.com, November 2008. [88] E. Munguia, T. Stephen, S. Intille, and K. Larson. Activity recognition in the home using simple and ubiquitous sensors. In Proceedings of the International Conference on Pervasive Computing, pages 158–175, 2004. [89] M. A. Musen. Dimensions of knowledge sharing and reuse. Comput. Biomed. Res., 25(5):435–467, 1992. [90] Myspace. http://www.myspace.com, November 2008. [91] R. Neches, R. Fikes, T. Finin, T. Gruber, R. Patil, T. Senator, and W. R. Swartout. Enabling technology for knowledge sharing. AI Mag., 12(3):36–56, 1991. [92] K. Nishigaki, K. Yasumoto, N. Shibata, M. Ito, and T. Higashino. Framework and rule- based language for facilitating context-aware computing using information appliances. In ICDCSW ’05: Proceedings of the First International Workshop on Services and Infrastructure for the Ubiquitous and Mobile Internet (SIUMI) (ICDCSW’05), pages 345–351, Washington, DC, USA, 2005. IEEE Computer Society. [93] C. Nugent, R. Davies, J. Hallberg, M. Donnelly, K. Synnes, M. Poland, J. Wallace, D. Finlay, M. Mulvenna, and D. Craig. HomeCI - A visual editor for healthcare professionals in the design of home based care. In 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pages 2787–2790, 2007. [94] C. Nugent, M. Donnelly, R. Bali, A. Dwivedi, T. Quinn, R. Naguib, and N. Black. Towards a web-based knowledge management system for cardiovascular disease. In Proceedings of EMBC2005, 2005. [95] C. Nugent, D. Finlay, R. Davies, M. Mulvenna, J. Wallace, C. Paggetti, E. Tamburini, and N. Black. The next generation of mobile medication management solutions. In- ternational Journal of Electronic Healthcare, 3:7–31, 2007. 180 Bibliography

[96] C. Nugent, D. Finlay, R. Davies, H. Wang, H. Zheng, J. Hallberg, K. Synnes, and M. Mulvenna. homeML - An Open Standard for the Exchange of Data Within Smart Environments. In 5th International Conference On Smart homes and health Telematics (ICOST), pages 121–129, 2007. [97] C. Nugent, P. McCullagh, E. McAdams, and A. Lymberis. Personalised Health Man- agement Systems: The Integration of Innovative Sesning, Textile, Information and Communication Technologies. Amsterdam: IOS Press, 2005. [98] C. D. Nugent, D. D. Finlay, P. Fiorini, Y. Tsumaki, and E. Prassler. Editorial home automation as a means of independent living. Automation Science and Engineering, IEEE Transactions on [see also Robotics and Automation, IEEE Transactions on], 5(1):1–9, 2008. [99] C. of the European Communities. Ageing well in the information society - an i2010 initiative. Action Plan on Information and Communication Technologies and Ageing Commission of the European Communities, COM(2007), 332:1–87, 2007. [100] Y. Ohmori, K. Ouchi, M. Hattori, and M. Doi. An xml based multimedia data acquisition and retrieval with wearable computers. In ICDCSW ’01: Proceedings of the 21st International Conference on Distributed Computing Systems, page 272, Washington, DC, USA, 2001. IEEE Computer Society. [101] Open ecg portal. http://www.openecg.net, November 2008. [102] Orkut. http://www.orkut.com, November 2008. [103] Osgi - the dynamic module system for java. http://www.osgi.org, November 2008. [104] OWL Web Ontology Language Overview. http://www.w3.org/TR/2004/REC-owl- features-20040210/, November 2008. [105] C. Paggetti and E. Tamburini. Remote management of integrated home care services: the dghome platform. In Proceedings of the 3rd International Conference on Smart homes and health Telematics, ICOST2005, pages 298–301, 2005. [106] P. Parnes. An IP-Multicast based Framework for Designing Synchronous Distributed Multi-User Applications on the Internet. PhD thesis, Luleå University of Technology, 1999. [107] P. Parnes, K. Synnes, and D. Schefström. mstar: Enabling collaborative applications on the internet. IEEE Internet Computing, 4(5):32–39, 2000. [108] R. Parviainen and P. Parnes. The mim web gateway to ip multicast e-meetings. In Proceedings of SPIE Conference on Multimedia Computing and Networking 2004, January 2004. Bibliography 181

[109] A. Pentland, T. Choudhury, N. Eagle, and P. Singh. Human dynamics: computation for organizations. Pattern Recogn. Lett., 26(4):503–511, 2005. [110] G. Perolle, P. Fraisse, M. Mavros, and I. Etxeberria. Automatic fall detection and activity monitoring for elderly. In Proceedings of MEDETEL, 2006. [111] T.-L. Pham, G. Schneider, S. Goose, and A. Pizano. Composite device computing environment: A framework for situated interaction using small screen devices. Personal and Ubiquitous Computing, 5(1):25–28, February 2001. [112] M. Philipose, K. P. Fishkin, M. Perkowitz, D. J. Patterson, D. Fox, H. Kautz, and D. Hahnel. Inferring activities from interactions with objects. IEEE Pervasive Com- puting, 3(4):50–57, 2004. [113] phpBB. http://www.phpbb.com/, November 2008. [114] Physionet, the research resource for complex physiologic signals. http://www.physionet.org, November 2008. [115] M. Pollack. Intelligent technology for an aging population: The use of ai to assist elders with cognitive impairment. AI Magazine, pages 9–24, 2005. [116] A. Pérez and R. Benjamins. Overview of knowledge sharing and reuse components: Ontologies and problem-solving methods. In Proceedings of the IJCAI-99 workshop on Ontologies and Problem-Solving Methods (KRR5), 1999. [117] A. Ranganathan, J. Al-Muhadi, and R. Campbell. Reasoning about uncertain contexts in pervasive computing environments. IEEE Pervasive Computing, 3(2):62–70, 2004. [118] A. Ranganathan and R. H. Campbell. An infrastructure for context-awareness based on ﬁrst order logic. Personal Ubiquitous Computing, 7(6):353–364, 2003. [119] M. Rettig. Prototyping for tiny ﬁngers. Commun. ACM, 37(4):21–27, 1994. [120] M. Román, Christopher, K. Hess, R. Cerqueira, A. Ranganathan, R. Campbell, and K. Nahrsted. Gaia: A Middleware Infrastructure to Enable Active Spaces. IEEE Pervasive Computing, pages 74–83, Oct-Dec 2002. [121] J. Rosen and B. Hannaford. Doc at a distance. IEEE Spectrum, 43(10):34–39, 2006. [122] A. Rugnone, C. Nugent, M. Donnelly, D. Craig, E. Vicario, C. Paggetti, and T. Tam- burini. HomeTL: A Visual Formalism, Based on Temporal Logic, for the Design of Home Based Care. In 3rd annual IEEE Conference on Automation Science and Engi- neering (CASE), pages 747–752, 2007. [123] Ruleml, the rule markup initiative. http://www.ruleml.org, November 2008. [124] N. Ryan. ConteXtML: Exchanging Contextual Information between a Mobile Client and the FieldNote Server . http://www.cs.kent.ac.uk/projects/mobicomp/fnc/ConteXtML.html, November 2008. 182 Bibliography

[125] B. Schilit, N. Adams, and R. Want. Context-aware computing applications. In In Proceedings of the Workshop on Mobile Computing Systems and Applications, pages 85–90. IEEE Computer Society, 1994. [126] B. Schilit, D. Hilbert, and J. Trevor. Context-aware communication. IEEE Wireless Communications, 9(5):46–54, October 2002. [127] B. N. Schilit and M. M. Theimer. Disseminating active map information to mobile hosts. IEEE Network, 8:22–32, 1994.

[128] W. N. Schilit. A system architecture for context-aware mobile computing. PhD thesis, Columbia University, 1995. [129] A. Schmidt. Ubiquitous Computing – Computing in Context. PhD thesis, Lancaster University, 2002. [130] H. Schulzrinne and E. Wedlund. Application layer mobility using sip. ACM Mobile Computing and Communications Review, 4:47–57, July 2000.

[131] L. Seligman and A. Rosenthal. Xml’s impact on databases and data sharing. Computer, 34(6):59–67, 2001. [132] G. Shafer. A mathematical theory of evidence. Princeton University Press, 1976. [133] Simple api for xml (sax). http://www.saxproject.org/, November 2008. [134] SIP: Session initiation protocol, rfc 2543. http://www.faqs.org/rfcs/rfc2543.html, 1999. [135] Skype. http://www.skype.com, November 2008. [136] J. R. Smith. Interoperable adaptive multimedia communication. IEEE Multimedia, 12(1):74–79, January–March 2005. [137] J. P. Sousa and D. Garlan. Aura: an architectural framework for user mobility in ubiquitous computing environments. In WICAS3: Proceedings of the IFIP 17th World Computer Congress - TC2 Stream / 3rd IEEE/IFIP Conference on Software Architec- ture, pages 29–43, Deventer, The Netherlands, The Netherlands, 2002. Kluwer, B.V. [138] T. Strang and C. Linnhoff-popien. A context modeling survey. In In: Workshop on Advanced Context Modelling, Reasoning and Management, UbiComp 2004 - The Sixth International Conference on Ubiquitous Computing, Nottingham/England, 2004.

[139] Teamspeak. http://www.goteamspeak.com/, November 2008. [140] K. Terﬂoth. Facts - a rule-based middleware architecture for wireless sensor networks. In In COMSWARE 2006: First IEEE International Conference on Communication System Software and Middleware, pages 1–8, 2006. Bibliography 183

[141] The Dynamic Groups Prototype. http://media.csee.ltu.se/projects/mymates/, Novem- ber 2008.

[142] The smart-its project. http://www.smart-its.org, November 2008. [143] D. Tran, D. Phung, H. Bui, and S. Venkatesh. A probabilistic model with parsimonious representation for sensor fusion in recognizing activity in pervasive environments. In Proceedings of the 18th International Conference on Pattern Recognition, 2006.

[144] V. Tuttlies, G. Schiele, and C. Becker. Comity - conﬂict avoidance in pervasive computing environments. In On the Move to Meaningful Internet Systems 2007: OTM 2007 Workshops, pages 763–772, 2007. [145] H. Wang, F. Azuaje, B. Jung, and N. Black. A markup language for electrocardiogram data acquisition and analysis (ecgml). BMC Medical Informatics and Decision Making, 3, 2003. [146] H. Wang, B. Rama, C. Chuah, R. Biswas, R. Gummadi, B. Hohlt, X. Hong, E. Kici- man, Z. Mao, J. Shih, L. Subramanian, B. Zhao, A. Joseph, and R. Katz. ICEBERG: An internet-core network architecture for integrated communications. IEEE Personal Communications, 7:10–19, Aug 2000. Special Issue on IP-based Mobile Telecommu- nication Networks. [147] Z. Wang and D. Garlan. Task-driven computing. Technical Report CMU-CS-00-154, School of Computer Science, Carnegie Mellon University, May 2000. [148] R. Want, A. Hopper, V. Falcão, and J. Gibbons. The active badge location system. ACM Transactions of Information Systems, 10(1):91–102, 1992. [149] M. Weiser. The computer for the 21st century. Scientiﬁc American, pages 94–104, September 1991. [150] W. H. O. (WHO). Adherence to Long-Term Therapies, Evidence for Action. 2003. [151] D. Wilson and C. Atkeson. Simultaneous tracking & activity recognition (star) using many anonymous, binary sensors. In In The Third International Conference on Pervasive Computing, pages 62–79. Springer-Verlag, 2005. [152] W. Woerndl. Requirements for personal information agents in the semantic web. Dis- tributed Applications and Interoperable Systems, 2003. 184 Appendix

185

Appendix A - Abbreviations

ADL Activities of Daily Living API Application Programming Interface DS Dempster-Shafer DSL Digital Subscriber Line ECG Electrocardiogram GPS Global Positioning System GUI Graphical User Interface HCI Human Computer Interaction HL7 Health Level Seven consortium HTTP Hypertext Transfer Protocol IP Internet Protocol JDK Java Development Kit MRSA Media Resource Selection Algorithm NFC Near Field Communication OWL Web Ontology Language PSTN Public Switched Telephone Network RFID Radio-frequency identification RMI Remote Method Invocation SIP Session Initiation Protocol SQL Structured Query Language UML Unified Modeling Language URI Uniform Resource Identifier URL Uniform Resource Locator WHO World Health Organization XML eXtended Markup Language

187 188 Appendix B - Glossary

Context Any information that can be used to characterize the situation of entities (i.e., whether a person, place or object) that are considered relevant to the interaction between a user and an application, including the user and the application themselves [1] Context-awareness The ability to sense, react, and adapt to changes in context. Media Resource A device capable of either capturing or presenting media (for example audio and video). A video camera is a media resource capable of capturing video, and a display connected to a computer is a media resource capable of presenting video. Mediated Communication Communication taking place over distance using different virtual media, such as audio and video. Pervasive computing Making many computers available throughout the physical envirnment, while making them effectively invisible to the user. Remote Healthcare The ability to provide healthcare services remotely, for example by utilising health monitoring systems, smart envri- onments, and mediated communication. Rich Communication Communication which includes at least audio and visual cues. Smart Environment A physical environment which is interwoven with devices which can sense, interact with, and present information to the envrionment. Ubiquitous computing Computing that is omnipresent and is, or appears to be, everywhere all the time; may involve many different computing devices that operate in the background.

189