Self-learning algorithms for the personalized interaction with people with dementia Bram Steenwinckel Supervisors: Prof. dr. ir. Filip De Turck, Dr. Femke Ongenae Counsellors: Dr. ir. Femke De Backere, Ir. Jelle Nelis Master's dissertation submitted in order to obtain the academic degree of Master of Science in Computer Science Engineering Department of Information Technology Chair: Prof. dr. ir. Bart Dhoedt Faculty of Engineering and Architecture Academic year 2016-2017 Self-learning algorithms for the personalized interaction with people with dementia Bram Steenwinckel Supervisors: Prof. dr. ir. Filip De Turck, Dr. Femke Ongenae Counsellors: Dr. ir. Femke De Backere, Ir. Jelle Nelis Master's dissertation submitted in order to obtain the academic degree of Master of Science in Computer Science Engineering Department of Information Technology Chair: Prof. dr. ir. Bart Dhoedt Faculty of Engineering and Architecture Academic year 2016-2017 PREFACE In the last couple of years, my interests in healthcare were enlarged by the several summer jobs I did, within this sector. It was there that I saw the need for more computational aids in the battle against various diseases. This same need is noticeable in the problem description of this dissertation. Being able to design possible solutions was, therefore, a privilege, honour and motivator. ”Self-learning algorithms for the personalized interaction with people with dementia” has been written in order to obtain the academic degree of Master of Science in Computer Science Engineering at the University of Ghent. I was engaged in researching and writing this dissertation from September 2016 till June 2017. After an intensive period of eight months, the last writings of this dissertation belongs to this note of thanks. During this journey of intensive learning, many people supported and helped me to achieve the findings stated in this master thesis. I think words of thanks are in order to these people. I would first like to thank my supervisors Prof. dr. ir. De Turck Filip and Dr. Ongenae Femke of the Departement of Information Technology at the University of Ghent. Prof. De Turck gave some valuable feedback for the different concepts in this dissertation and enlarged my knowledge in computer science during my full master program. Dr. Ongenae was always available whenever I had a question about my research and steered me in the right the direction whenever needed during multiple interactive sessions. I would also like to thank the researchers who gave some additional knowledge about the different concepts discussed in this dissertation: Ir. Bohez Steven, Ir. Mahieu Christof and Ir. Nelis Jelle, who helped with the fundamental concepts of reinforcement learning, the Nao robotic interactions and the design of external sensors. Also, I would like to thank Schaballie Jeroen and De Pestel Stijn. My research would not have been possible without their help. I would particularly like to single out my counsellor Dr. ir. De Backere Femke, I want to thank you for all the provided feedback and the encouraging words when needed. I would also like to acknowledge a friend, Joris Heyse, who tested my designed application. We were not only able to support each other by deliberating over our problems and findings on reinforcement learning techniques but also happily by talking about things other than just our papers. Finally, I must express my very profound gratitude to my parents, sister Lien and my lovely Laura for providing me with unfailing support and continuous encouragement throughout my years of study and through the process of researching and writing this dissertation. This accomplishment would not have been possible without you. Thank you. Bram Steenwinckel i The author gives permission to make this master dissertation available for consultation and to copy parts of this mas- ter dissertation for personal use. In the case of any other use, the copyright terms have to be respected, in particular with regard to the obligation to state expressly the source when quoting results from this master dissertation. Gent, 2 July 2017. ii Self-learning algorithms for the personalized interaction with people with dementia Supervisors: Prof. dr. ir. Filip De Turck, Dr. Femke Ongenae Counsellors: Dr. ir. Femke De Backere, Ir. Jelle Nelis Department of Information Technology Chair: Prof. dr. ir. Bart Dhoedt Faculty of Engineering and Architecture Academic year 2016-2017 Abstract: The number of people with dementia (PwD) residing in nursing homes (NH) increases rapidly. Behavioural disturbances (BDs) such as wandering and aggressions are the main reasons to hospitalise these people. Social robots could help to resolve these BDs by performing simple interactions with the patients. The WONDER project investi- gates the necessary functionality to have such a robot autonomously walking from one resident to another, each time engaging in a personalised interaction. This paper examines whether self-learning algorithms could be designed to select the robotic interactions, preferred by the patients, during these interventions. K-armed bandit algorithms were compared in simulated environments for single and multiple patients to find the beneficial learning agents and ac- tion selection policies. The single patient tests show the advantages of selecting actions according to an UCB policy, while the multi-patient tests analyse the benefits of using additional, contextual information. Afterwards, the learn- ing application was provided with a framework to operate in more realistic situations. Tests with real PwD will still be needed before this designed learning application can be integrated within the full WONDER system. Keywords: Robot-Assisted Intervention, People with Dementia, Personalised interaction, Bandit algorithms iii Self-learning algorithms for the personalized interaction with people with dementia Bram Steenwinckel Supervisors: Prof. dr. ir. Filip De Turck, Dr. Femke Ongenae Counsellors: Dr. ir. Femke De Backere, Ir. Jelle Nelis Abstract— The number of people with dementia (PwD) resid- into the daily care processes for the prevention and alleviation ing in nursing homes (NH) increases rapidly. Behavioural dis- of BDs. WONDER will research the necessary functionality turbances (BDs) such as wandering and aggressions are the main to have the robot autonomously walking from one resident to reasons to hospitalise these people. Social robots could help to another, each time engaging in a personalised interaction, for resolve these BDs by performing simple interactions with the pa- tients. The WONDER project investigates the necessary func- example, playing a favourite song or asking questions about tionality to have such a robot autonomously walking from one memorable events in the lifetime of the PwD [10]. The main resident to another, each time engaging in a personalised inter- idea behind these interactions is that Zora will generate stim- action. This paper examines whether self-learning algorithms uli to elicit personal memories with associated positive feelings could be designed to select the robotic interactions, preferred by that have calming and reassuring effects onto these PwD. the patients, during these interventions. K-armed bandit algo- rithms were compared in simulated environments for single and multiple patients to find the beneficial learning agents and action selection policies. The single patient tests show the advantages of selecting actions according to an UCB policy, while the multi- patient tests analyse the benefits of using additional, contextual information. Afterwards, the learning application was provided with a framework to operate in more realistic situations. Tests with real PwD will still be needed before this designed learning application can be integrated within the full WONDER system. Keywords— Robot-Assisted Intervention, People with Demen- tia, Personalised interaction, Bandit algorithms I. Introduction Worldwide, almost 44 million people have dementia-related diseases where it is the most common in Western Europe [1]. Approximately 43% of these people with dementia (PwD) are staying at nursing homes (NH), rising to 76% of those with ad- vanced dementia [2]. Beside the amnesia, all these PwD suffer Fig. 1. Conceptual architecture of the WONDER project from so-called behavioural disturbances (BDs) like mood dis- orders, hallucinations, wandering and aggressions. Pharmaco- The WONDER system will let the care coordinator create logical interventions are used only for acute situations in the profiles with personal information about the residents together management of these BDs because these treatments do not ad- with organisational data, such as NH map and activity timeta- dress the underlying psychosocial reasons and may have adverse bles. Combined with information provided by the available sen- side effects [3]. Many different non-pharmacological therapies sors in the NH, and the customised wearables of the patients, a are designed to resolve specific BDs by interacting with the per-resident intervention strategy is determined. In acute situ- PwD and without the harmful effects of medical interventions ations, the robot can be sent immediately to distract the PwD [4]. However, these therapy sessions are more time consuming, temporarily. Pro-active interventions are scheduled collabora- and due to the increased strain on the available resources within tively over multiple robots throughout the day and night, taking healthcare, many NH avoid these therapies. into account the limitations