Serious Games for Health and Medicine. a Cardiopulmonary Resuscitation (Cpr) Case Study

SERIOUS GAMES FOR HEALTH AND MEDICINE. A CARDIOPULMONARY RESUSCITATION (CPR) CASE STUDY

Voravika WATTANASOONTORN

Dipòsit legal: Gi. 1775-2013 http://hdl.handle.net/10803/128270

ADVERTIMENT. L'accés als continguts d'aquesta tesi doctoral i la seva utilització ha de respectar els drets de la persona autora. Pot ser utilitzada per a consulta o estudi personal, així com en activitats o materials d'investigació i docència en els termes establerts a l'art. 32 del Text Refós de la Llei de Propietat Intel·lectual (RDL 1/1996). Per altres utilitzacions es requereix l'autorització prèvia i expressa de la persona autora. En qualsevol cas, en la utilització dels seus continguts caldrà indicar de forma clara el nom i cognoms de la persona autora i el títol de la tesi doctoral. No s'autoritza la seva reproducció o altres formes d'explotació efectuades amb finalitats de lucre ni la seva comunicació pública des d'un lloc aliè al servei TDX. Tampoc s'autoritza la presentació del seu contingut en una finestra o marc aliè a TDX (framing). Aquesta reserva de drets afecta tant als continguts de la tesi com als seus resums i índexs.

ADVERTENCIA. El acceso a los contenidos de esta tesis doctoral y su utilización debe respetar los derechos de la persona autora. Puede ser utilizada para consulta o estudio personal, así como en actividades o materiales de investigación y docencia en los términos establecidos en el art. 32 del Texto Refundido de la Ley de Propiedad Intelectual (RDL 1/1996). Para otros usos se requiere la autorización previa y expresa de la persona autora. En cualquier caso, en la utilización de sus contenidos se deberá indicar de forma clara el nombre y apellidos de la persona autora y el título de la tesis doctoral. No se autoriza su reproducción u otras formas de explotación efectuadas con fines lucrativos ni su comunicación pública desde un sitio ajeno al servicio TDR. Tampoco se autoriza la presentación de su contenido en una ventana o marco ajeno a TDR (framing). Esta reserva de derechos afecta tanto al contenido de la tesis como a sus resúmenes e índices.

WARNING. Access to the contents of this doctoral thesis and its use must respect the rights of the author. It can be used for reference or private study, as well as research and learning activities or materials in the terms established by the 32nd article of the Spanish Consolidated Copyright Act (RDL 1/1996). Express and previous authorization of the author is required for any other uses. In any case, when using its content, full name of the author and title of the thesis must be clearly indicated. Reproduction or other forms of for profit use or public communication from outside TDX service is not allowed. Presentation of its content in a window or frame external to TDX (framing) is not authorized either. These rights affect both the content of the thesis and its abstracts and indexes. DOCTORAL THESIS

Serious Games for Health and Medicine. A Cardiopulmonary Resuscitation (CPR) Case Study.

Author:

Voravika WATTANASOONTORN

2013 1

DOCTORAL T HESIS

Serious Games for Health and Medicine. A Cardiopulmonary Resuscitation (CPR) Case Study.

Author: Voravika WATTANASOONTORN

2013

Programa de Doctorat en Tecnologia

Advisors: Prof.Dr. Mateu SBERT Dra. Imma BOADA

Presented in partial fulﬁlment of the requirements for a doctoral degree from the University of Girona

Acknowledgment

First of all, I would like to express my sincere gratitude to my lovely advisor Prof.Dr. Mateu Sbert and co-advisor Dra. Imma Boada for the continuous support of my Ph.D study and research, for their patience, motivation, enthusiasm, and immense knowledge. Their guidance helped me all the time of research and writing of this thesis. I could not have imagined having a better advisor and mentor for my Ph.D study. My sincere thanks also goes to Prince of Songkla University, particularly the faculty of technology and environment, for the grant and all supports they have provided to me. I would like to thank my fellow colleagues in Graphics & Imaging Laboratory (GIlab): Dr. Rubén Jesús García Hernández, Milán Magdics, Carles Blavi, Juan Manuel Garcia Gonzalez, Antonio Rodriguez Benitez, Dr. Dolors Juvinyá Canal and Dr. Josep Olivet Pujol for their kind cooperation in the LISSA project. As well as to my labmates: Xavier Bonaventura Brugues, Dr. Roger Bramon, Márius Vila, Ester Bonmatí, Olga Petrenko and Nicolau Sunyer for their help and shares. Many thanks also go to Dr. Miquel Feixas, Dr. Anton Bardera, Dr. Roel Martinez, Dr. Alessandro Artusi, Rosa Teixidor, Cristina Rubirola and Montse Bragulat for their kind advices during the time I studied in UdG; as well as to José Burrel, Sakib Jalil, Busquets and Centeno families, lovely Catalan people, for their friendships. My appreciation cannot be completed without mentioning endless effort put in place by my beloved husband, Kitt - Kittipong Srikimkaew. Last but not the least,

I would like to thank my parents (Vorasak and Manavika) as well as Kitt0s parents (Prasit and Pornphan), for supporting us spiritually throughout our life.

5 Grant Support

This work has been supported by the research projects TIN2010-21089-C03-01, IPT-2011-1516-370000 and IPT-2011-0885-430000 (Spanish Commission for Science and Technology), by grant 2009SGR643 (Catalan Government) and by mobility grant PSU064/1071 from the Prince of Songkla University, Thailand.

Related Publications

The work of this thesis has lead to the following publications:

V.Wattanasoontorn, R. García, I. Boada, M. Sbert, Serious games for e-healthcare, • The proceeding of Asian-European Workshop on Serious Game and Simulation, 25th Annual Conference on Computer Animation and Social Agents (CASA 2012), Singapore, 9-11 May 2012 and Springer Book of Simulations, Serious Games and Their Applications, Series: Gaming Media and Social Effects, ch.10, October 31, 2013.

V. Wattanasoontorn, I. Boada, M. Sbert, The framework of a LIfe Support Simula- • tion Application, The proceedings of 4th International Conference on Games and Virtual Worlds for Serious Applications (VS GAMES12), 29Oct to 1Nov, 2012. Genoa, Italy and the journal of Procedia Computer Science, Volume 15, 2012, Pages 293 - 294, ISSN 1877-0509

V. Wattanasoontorn, I. Boada, M. Sbert, LISSA: A Serious Game to learn Car- • diopulmonary Resuscitation, The proceedings of the 8th international conference on Foundations of Digital Games (FDG2013): Games for Learning workshop, 14 -17 May 2013, Chania, Crete, Greece.

V. Wattanasoontorn, M. Magdics, I. Boada, M. Sbert, Visual realism in 3D serious • games for learning: A case study, The proceedings of 15th International Con- ference on Enterprise Information Systems: Workshop on Interaction Design in Educational Environments (ICEIS 2013), July 5, 2013, Angers, France.

V. Wattanasoontorn, M. Magdics, M. Sbert, A Kinect-based System for Cardiopul- • monary Resuscitation Simulation: A pilot study, The proceedings of 4th Inter- national Conference on Serious Games Developments and Applications (SGDA 2013), 25 - 27 Sep, 2013, Trondheim, Norway and will be published in the Springer Lecture Notes in Computer Science (LNCS) series LNCS 8101.

V.Wattanasoontorn, R. García, I. Boada, M. Sbert, Serious games for health, Enter- • tainment Computing Journal, Elsevier, Ref. No.: ENTCOM-D-13-00008R1 [accepted 23 Sep 2013]. V. Wattanasoontorn, I. Boada, M. Sbert, LISSA, a new e-learning environment to • learn CPR, which has been submitted to the Journal of Resuscitation, Elsevier, 2013 [submitted].

9 Abbreviations and Notation

µ Mean σ Standard deviation 2D Two dimensions 3D Three dimensions ABCs Airway, breathing and circulation AED Automated external deﬁbrillator AHA The American heart association ASD Autism Spectrum Disorder AUD Alcohol use disorder Cog Cognitive skills CPR Cardiopulmonary resuscitation CPR (30:2) Perform CPR 30 chest compressions, at a rate of 100 - 120 per minute and give 2 ventilations to the patient ERC European resuscitation council FA Focused on health acquisition and medical skills FE Focused on entertainment FH Focused on Health GUI Graphic user interface HMDs Head-mounted displays LISSA LIfe support simulation application LS Life support MCH Maternal-child health MMORPG Massively multiplayer online role playing game Mot Motor skills NPR Non-photorealistic O/S Over shoulder (Camera view) PCs Personal computers PN Practical nurses PR Photorealistic RPG Role play game VR Virtual Reality WHO World health organization

List of Figures

1.1 Maslow pyramid...... 8

2.1 The differences between games and serious games functionalities . . . . 12 2.2 Core components of serious games ...... 13 2.3 Dale cone of learning [41], which shows the degree or levels of learning that occurs when learners engage to each learning style (reading, hearing, seeing, watching, participating and doing) ...... 16 2.4 Example of serious games for learning ...... 17 2.5 Polygons used to model a hospital scene ...... 19 2.6 The hospital scene with textured polygons ...... 19 2.7 The hospital scene after lighting applied ...... 20 2.8 A same scene illuminated with different light sources ...... 20 2.9 A realistic rendering of the hospital scene ...... 21 2.10 The hospital scene rendered using different not-photorealistic styles . . . 21 2.11 Type of realism in games, with regard to [69] and [129] ...... 22 2.12 Club head speed function of three different golf games ...... 22 2.13 Screen-shots of three soccer games which have different level of realism, increasing from left to right ...... 23 2.14 Visual realism components ...... 24 2.15 Different character proportion ...... 24 2.16 Comparison of vector-based and bitmap-based textures ...... 25 2.17 Comparison of fantastic light (Hunted: The Demon’s Forge) and realistic light (Call of Duty) ...... 25 2.18 Examples of non-photorealistic rendering (NPR) applied to a game scene 26 2.19 The ten render styles ranking from abstract to realistic [131] ...... 26 2.20 VRDTS (Virtual Reality Dental Training System), the dental training simulator that utilizes a 3 dimensional haptic interface to enhance the physical realism while playing [158] ...... 27

3.1 Numbers of surveyed serious games for health according to the year of publication ...... 30 3.2 Classiﬁcations of serious games for health ...... 42 3.3 Serious games purpose related to stage of disease ...... 45 3.4 Classiﬁcation of serious games for health by player ...... 46 3.5 (a) Distribution of surveyed serious games according to type of players and (b) number of surveyed serious games for patients and non-patients according to the year of publication ...... 55 3.6 (a) Distribution of surveyed serious games according to its objective . . . 56 12 List of Figures

3.7 (a) Distribution of surveyed serious games focused on learning according to its objective, medical care or learning and (b) number of surveyed serious games focused on learning objective according to the year of publication...... 56 3.8 Distribution of surveyed serious games with regard to disease ...... 56 3.9 (a) Distribution of surveyed serious games with regard to brain related disease and (b) number of surveyed serious games focused on brain disease according to the year of publication ...... 57 3.10 (a) Distribution of surveyed serious games according to its purpose, (b) distribution of surveyed serious games for patients according to its purpose and (c) distribution of surveyed serious games for non-patients according to its purpose ...... 58 3.11 (a) Distribution of surveyed serious games with regard to application area, (b) distribution of surveyed serious games for patients with regard to application area and (c) distribution of surveyed serious games for non-patients with regard to application area ...... 58 3.12 Distribution of surveyed serious games according to interaction tool . . . 59 3.13 Distribution of surveyed serious games according to top four interaction tools and the year of publication ...... 59 3.14 (a) Distribution of surveyed serious games with regard to interface, (b) distribution of surveyed serious games for patients with regard to interface and (c) distribution of surveyed serious games for non-patients with regard to interface ...... 60 3.15 Number of 2D and 3D interfaces of our surveyed serious games according to the year of publication ...... 60 3.16 (a) Distribution of surveyed serious games with regard to its number of players, (b) distribution of surveyed serious games for patients with regard to number of players and (c) distribution of surveyed serious games for non-patients with regard to number of players ...... 61 3.17 (a) Distribution of surveyed serious games with regard to its genre, (b) distribution of surveyed serious games for patients with regard to game genre and (c) distribution of surveyed serious games for non-patients with regard to game genre ...... 61 3.18 Distribution of surveyed serious games according to (a) adaptability, (b) progress monitoring, (c) performance feedback and (d) portability . . . . . 62 3.19 Distribution of surveyed serious games according to game engine . . . . . 62 3.20 (a) Distribution of surveyed serious games with regard to game platform, (b) distribution of surveyed serious games for patients with regard to game platform and (c) distribution of surveyed serious games for non- patients with regard to game platform ...... 63 3.21 (a) Distribution of surveyed serious games with regard to connectivity, (b) distribution of surveyed serious games for patients with regard to connectivity and (c) distribution of surveyed serious games for non- patients with regard to connectivity ...... 63 List of Figures 13

4.1 CPR guideline provided by the European Resuscitation Council (ERC2010) 67 4.2 CPR with automated external defibrillation (AED) guideline provided by the European Resuscitation Council (ERC2010) ...... 68 4.3 Main modules of LISSA framework ...... 70 4.4 (a) CPR flowchart for an adult patient [63] and (b) proposed UML state machine to model the CPR flowchart ...... 72 4.5 UML state machine of basic cpr procedure (without AED machine) . . . 73 4.6 Some of the models stored in the LISSA database and used to create CPR scenes: (a) football field, (b) ambulance, (c) hospital, (d) helper and victim models and (e) some of the supported devices...... 74 4.7 Interface designed to determine the parameters that define the CPR scene 76 4.8 CPR scenarios automatically created with the objects from the content creation database of LISSA ...... 76 4.9 Action menu of LISSA ...... 77 4.10 Some of the CPR procedures that can be performed through the LISSA game ...... 78 4.11 Glyphs over the patient to show the user interaction areas ...... 78 4.12 The LISSA performance indicator that informs user in real-time . . . . . 79 4.13 LISSA performance report of a problem ...... 79 4.14 LISSA graphic user interface for the instructor ...... 82 4.15 LISSA graphic user interface for the learner ...... 83 4.16 Problem solving ...... 85

5.1 Relationship between visual realism elements and the 3D production process ...... 88 5.2 Human character proportion in LISSA ...... 89 5.3 Number of polygons in ambulance scene ...... 90 5.4 Difference between low-polygon model and high polygon model . . . . . 91 5.5 Some of the texture maps used for a LISSA character ...... 92 5.6 Some of the textures used for the automatic deﬁbrillator machines . . . 92 5.7 Example of high precision animation while two objects have touched . . 94 5.8 Viewpoint ...... 94 5.9 Examples of NPR rendering for pre-selection ...... 96 5.10 Comparison of PR and NPR to LISSA ...... 97 5.11 Comparison of 1st and 3rd person viewpoint in different actions . . . . . 97 5.12 Evaluation of average results (in red); error polylines (green and blue) at one standard deviation. R, VP and L denote questions related to rendering mode, view point and the effectiveness of LISSA, respectively. . . 100

6.1 Optical-based motion tracking system with markers (left) and marker- free (right) ...... 105 6.2 Microsoft Kinect ...... 106 6.3 Correct CPR chest compression sequence presented in LISSA ...... 107 6.4 Example of correct and incorrect arm position ...... 109 14 List of Figures

6.5 Some of the CPR criteria that can be performed through the LISSA game 110 6.6 The integration of Kinect into LISSA system ...... 110 6.7 (a) Screenshot of LISSA based on Kinect connection system; (b) the CPR- chest compression rate information and (c) the arm position tracking symbol - green means correct position and red means incorrect position 111 6.8 Experiment of CPR-chest compression on manikin and obtaining the chest compression data from LISSA, iCPR and Laderal PC Skillreport- ing System ...... 112

7.1 Major components of instruction [64] ...... 116 7.2 Bloom0s taxonomy ...... 117 7.3 (a) Electronic CPR Monitor [83] and (b) Resusci Anne Simulator, a full- body CPR training manikin [112] ...... 119 7.4 Flow diagram of experiment procedure for learner ...... 122 7.5 Exam questions used in our experiment ...... 123 7.6 Average of evaluation results (in red); error polylines (green and blue) at one standard deviation with respect to teaching usability ...... 125 7.7 The learning outcomes of differences CPR learning styles (A-E) ...... 126 7.8 (Top) The improvement of learning outcome for the participant in G1 and (Bottom) The improvement of learning outcome for the participant in G2 ...... 128 7.9 Average of evaluation results (in red); error polylines (green and blue) at one standard deviation with respect to learning usability ...... 130 List of Tables

3.1 Review of serious games ...... 31 3.2 Classiﬁcation and comparison of health games from our survey ...... 49

4.1 Comparison of CPR applications ...... 69

5.1 Characteristics of sample for the questionnaire ...... 98 5.2 Comparison results of photorealistic (PR) and non-photorealistic rendering (NPR) ...... 99 5.3 Comparison results of 1st and 3rd person viewpoint ...... 99 5.4 Visual realism results of LISSA ...... 101

6.1 Required information with regard to CPR chest compression criteria (where absolute means that we have to obtain the given parameter in absolute measures and relative denotes that it has to be known only with respect to another object...... 108 6.2 Comparison of different motion tracking performance with respect to [244], Note: last column presents only devices compatible with CPR actions in our platform...... 108 6.3 Experiment result of CPR chest compression rate (times per minute) and correct arm position tracking capability for each application. More details are explained in Section 6.6) ...... 112

7.1 Bloom0s taxonomy, learning objective categorization [21] ...... 117 7.2 Main functionalities of LISSA ...... 120 7.3 Functions for the outside of hospital problems ...... 121 7.4 Teaching assessment of LISSA ...... 125 7.5 Examination score (where A is pre-examination score, B is post-examination of mannequin, C is post-examination of LISSA after B, D is post-examination of LISSA and E is post-examination of mannequin after D) ...... 127 7.6 Learning assessment of LISSA ...... 129

Contents

Abstract1

1 Introduction7 1.1 Motivation ...... 7 1.2 Objectives...... 8 1.3 Thesis outline ...... 9

2 Background 11 2.1 Introduction ...... 11 2.2 Serious games...... 11 2.3 Core components of a serious game...... 12 2.4 Serious games as an e-learning tool...... 16 2.5 3D Production for games ...... 18 2.6 Realism in games ...... 22 2.6.1 Functional realism ...... 22 2.6.2 Visual realism ...... 23 2.6.3 Physical realism ...... 26 2.7 Conclusion ...... 28

3 Serious games for health 29 3.1 Introduction ...... 29 3.2 The use of serious games to promote health ...... 29 3.3 Method and scope of study ...... 30 3.4 Review of serious games for health ...... 31 3.5 Classification of surveyed serious games for health ...... 42 3.5.1 Classification by game purpose ...... 42 3.5.2 Classification by functionality ...... 43 3.5.3 Classification by stage of disease ...... 44 3.5.4 Classification by player wellness (patients/non-patients) . . . . . 46 3.6 Results ...... 48 3.7 Discussion ...... 55 3.8 Conclusion ...... 64

4 LISSA, a Life Support Simulation Application 65 4.1 Introduction ...... 65 4.2 Related work ...... 65 4.2.1 The basic CPR ﬂowchart ...... 65 4.2.2 Serious games and CPR ...... 66 4.3 The LISSA framework ...... 69 4.3.1 The main components ...... 70 18 Contents

4.3.2 The content creation module ...... 71 4.3.3 Correction Module ...... 77 4.4 Results ...... 80 4.4.1 LISSA interfaces ...... 80 4.4.2 Implementation details ...... 81 4.5 Conclusion ...... 86

5 Visual realism design for LISSA 87 5.1 Introduction ...... 87 5.2 Background ...... 87 5.3 Visual realism within LISSA ...... 88 5.3.1 Form ...... 88 5.3.2 Textures ...... 90 5.3.3 Illumination ...... 93 5.3.4 Movement ...... 93 5.3.5 Viewpoint ...... 94 5.4 Visual realism and user preferences ...... 95 5.5 Results ...... 98 5.6 Conclusion ...... 99

6 Kinect-based system for LISSA 103 6.1 Introduction ...... 103 6.2 Background ...... 103 6.2.1 Human motion tracking system ...... 104 6.2.2 Kinect ...... 105 6.3 Human motion tracking tool for CPR ...... 107 6.4 Kinect-based system for LISSA ...... 108 6.4.1 Arm position ...... 108 6.4.2 Counting compressions ...... 109 6.4.3 Other CPR factors ...... 109 6.4.4 Implementation details ...... 109 6.5 Results ...... 110 6.6 Discussion and limitations ...... 113 6.7 Conclusion ...... 114

7 Evaluation of LISSA as CPR teaching/learning environment 115 7.1 Introduction ...... 115 7.2 Background ...... 116 7.2.1 Instruction components ...... 116 7.2.2 Manikin based CPR training ...... 119 7.3 LISSA as teaching/learning environment ...... 120 7.4 Evaluation ...... 121 7.4.1 Teaching evaluation ...... 121 7.4.2 Learning evaluation ...... 122 Contents 19

7.5 Result ...... 124 7.5.1 Teaching performance ...... 124 7.5.2 Learning performance ...... 125 7.6 Conclusion ...... 129

8 Conclusion 131 8.1 Contribution and publication ...... 131 8.2 Future work ...... 133

Bibliography 135

Abstract

Today, the term serious game is becoming increasingly popular and its study is a recent practice, within less than two decades. In spite of the similarities between computer games and serious games there are many differences between them, particularly on the intention of giving specific experience, knowledge and skills to the audiences. Serious games are used in many areas of knowledge, including military, health, manufacturing, education and medicine. In this thesis we will consider serious games for health. Also, we will focus on to the study and development of new techniques capable to improve three main aspects involved in the development of a serious game. The first one is the teaching/learning component of the game which relates with the capability of the game to transmit some predefined skills to the user. The second one is the graphic realism which influences user perception on the game. The last one is the user interaction which aims improving the physical realism while playing. As a case study we will consider the cardiopulmonary resuscitation (CPR) protocol. CPR is a first aid key survival technique used to stimulate breathing and keep blood flowing to the heart. Its effective administration can significantly increase the chances of survival for victims of cardiac arrest that take place outside of hospital. We will present a survey of serious games for health and medicine that have been proposed in the last decade, proposing new classifications of these games with the idea to make its evaluation easier. We will present LISSA, a serious game to teach and learn CPR. LISSA actions will turn around a CPR scenario that reproduces an emergency situation that requires CPR procedures. The CPR scenario will be defined by the instructor and presented to the learner as problem. The learner will solve this problem applying the CPR procedures in a game mode. In the context of LISSA, we will study the visual realism focusing on 3D serious games and their elements, evaluating photo and non-photo realistic effects, as well as, camera viewpoint comparing 1st and 3rd person view. In addition, we will study the physical realism in order to improve the interaction between game and player. We will present a new Kinect-based strategy capable to reproduce important factors of the CPR protocol such as compression rhythm, chest position, compression depth, hand position and breath giving. Finally, we will evaluate LISSA in a real scenario focusing on CPR skill improvement for learner and usability for the instructor.

Resum

El joc seriós, de l0anglès serious games, s0ha convertit en un terme popular i cada vegada més estudiat. Malgrat les similituds entre els jocs d0ordinador i els jocs seriosos, cal dir que hi ha també moltes diferències entre ells, la principal és la ﬁnalitat del joc. El joc seriós té com a objectiu donar una experiència especíﬁca, coneixements i habilitats al jugador. Els àmbits d0aplicació d0aquests jocs són molts i variats, entre d0altres, el militar, la salut, la indústria, l0educació i la medicina. En aquesta tesi considerarem els jocs seriosos per a la salut.

Ens centrarem en l0estudi i desenvolupament de noves tècniques que permetin millorar tres aspectes principals que intervenen en el desenvolupament d0un joc seriós. El primer és el component d0ensenyament/aprenentatge del joc que es relaciona amb la capacitat del joc per transmetre algunes habilitats predefinides. El segon és el realisme gràfic que influeix en la percepció de l0usuari en el joc. El tercer és la interacció de l0usuari que té com a objectiu millorar el realisme físic durant el joc. Com a cas d0estudi considerarem el protocol de reanimació cardiopulmonar (RCP). L0RCP és una tècnica de primers auxilis utilitzada per estimular la respiració i fer que la sang flueixi cap al cor. La seva administració eficaç pot augmentar significativament les possibilitats de supervivència de les víctimes d0una aturada cardíaca fora de l0hospital. En el marc de la tesi presentarem un estudi dels jocs seriosos per a la salut i la medicina proposats en aquesta última dècada, proposant noves classificacions d0aquests jocs amb la idea de fer més fàcil la seva avaluació. Presentarem també LISSA , un joc seriós per ensenyar i aprendre RCP.Totes les accions d0aquest joc giren al voltant d0una situació d0emergència que requereixi maniobres de RCP. L0escenari de RCP es definit per un instructor i presentat a l0alumne com un problema. L0alumne resoldrà aquest problema aplicant els procediments de RCP en forma de joc. En el context del LISSA, estudiarem el realisme visual dels jocs seriosos 3D i els seus elements, avaluarem els efectes fotorealistes i no-fotorealistes i la posició de la càmera en primera i tercera persona. A més, estudiarem el realisme físic per tal de millorar la interacció entre el joc i el jugador. Proposarem una nova tècnica basada en Kinect capaç per reproduir factors clau del protocol RCP com ara el ritme de compressió, la posició al pit, la profunditat, la posició de les mans durant la compressió i el boca a boca. Finalment, avaluarem LISSA en un escenari real centrant-nos en la millora de les habilitats de RCP per part de l0alumne i la usabilitat per part de l0instructor.

Resumen

El juego serio, del inglés serious games, se ha convertido en un término popular y cada vez más estudiado. A pesar de las similitudes entre los juegos de ordenador y los juegos serios, hay que decir que hay también hay muchas diferencias entre ellos, la principal es la finalidad del juego. El juego serio tiene como objetivo dar una experiencia específica, conocimientos y habilidades al jugador. Los ámbitos de aplicación de estos juegos son muchos y variados, entre otros, el militar, la salud, la industria, la educación y la medicina. En esta tesis consideraremos los juegos serios para la salud. Nos centraremos en el estudio y desarrollo de nuevas técnicas que permitan mejorar tres aspectos principales que intervienen en el desarrollo de un juego serio. El primero es el componente de enseñanza/aprendizaje del juego que se relaciona con la capacidad del juego para transmitir algunas habilidades predefinidas. El segundo es el realismo gráfico que influye en la percepción del usuario en el juego. El tercero es la interacción del usuario que tiene como objetivo mejorar el realismo físico durante el juego. Como caso de estudio consideraremos el protocolo de reanimación cardiopulmonar (PRC). El PRC es una técnica de primeros auxilios utilizada para estimular la respiración y hacer que la sangre fluya hacia el corazón. Su administración eficaz puede aumentar significativamente las posibilidades de supervivencia de las víctimas de un paro cardiaco fuera del hospital. En el marco de la tesis presentaremos un estudio de los juegos serios para la salud y la medicina propuestos en esta última década, proponiendo nuevas clasificaciones de estos juegos con la idea de hacer más fácil su evaluación. Presentaremos también LISSA, un juego serio para enseñar y aprender PRC. Todas las acciones de este juego giran alrededor de una situación de emergencia que requiera maniobras de PRC. El escenario de PRC es definido por un instructor y presentado al alumno como un problema. El alumno resolverá este problema aplicando los procedimientos de PRC en forma de juego. En el contexto del LISSA, estudiaremos el realismo visual de los juegos serios 3D y sus elementos, evaluaremos los efectos fotorealistes y no-fotorealistes y la posición de la cámara en primera y tercera persona. Además, estudiaremos el realismo físico para mejorar la interacción entre el juego y el jugador. Propondremos una nueva técnica basada en Kinect capaz de reproducir factores clave del protocolo PRC como son el ritmo de compresión, la posición al pecho, la profundidad, la posición de las manos durante la compresión y el boca a boca. Finalmente, evaluaremos LISSA en un escenario real centrándonos en la mejora de las habilidades de PRC por parte del alumno y la usabilitat por parte del instructor.

CHAPTER 1 Introduction

1.1 Motivation

The term Serious Games was introduced by David Rejeski and Ben Sawyer, in their white paper Serious Games Initiative (2002), as entertaining games with non entertainment goals [197]. Since this first definition, many different ones have been proposed. Amongst them, Michael and Chen [138] defined serious games as games that educate, train and inform. Zyda [246] defined serious games as a mental contest, played with a computer in accordance with specific rules, that uses entertainment to further government or corporate training, education, health, public policy, and strategic communication objectives. Although there is no single definition of the serious game concept, all the proposed definitions convey the same idea which is using games to teach or transmit something. This has lead some analysts to describe them as the next wave of technology-mediated learning [9]. Serious games are present in many areas, including military, health, business, social, science, manufacturing, medical, education and training. In the military context, serious games are used to train pilot or improve the command performance and identify, on vehicles or traps, speed of player [29][212][229]. For the education and training, which is the most farthest used, serious games are applied to pass on the knowledge or expertise through the games which can be adopted into various purposes, e.g. for career training [49][48]. In the manufacturing areas, most of the serious games are focused on risk awareness in the working area [61][209] and maintenance support [4]. In the health and medical context, serious games are designed to educate and train health care professionals to avoid medical errors [84][107] or in rehabilitation processes, to reproduce the repetitive tasks that have to be done by the patients [47][177]. In this thesis we will focus on health and medical area. Health is one of the main issues that affects people in every stage of life (from infancy to old age). In Maslow hierarchy of needs [130], see Figure 1.1, health is represented in the second lowest level after basic needs required for life are fulfilled. Health is then a very basic need, and maintaining health is a priority. Additionally, the desirable human characteristics located in higher levels of Maslow pyramid, which are needed for a functioning, peaceful society, are negatively impacted by lack of health. We will focus our research on serious games for health, in particular for cardiopulmonary resuscitation (CPR). CPR is a first aid key survival technique used to stimulate breathing and keep blood flowing to the heart. Its effective administration can significantly increase the chances of survival for victims of cardiac arrest that take place outside of hospital. Since 1960, when Kouwenhoven published an article stating that 8 Chapter 1. Introduction

Figure 1.1: Maslow pyramid anyone, anywhere, could perform CPR [108], providing CPR has become an essential competency not only for expert or professional but also for laypersons. The development of a serious game is challenging. On the one hand, it requires to integrate the serious contents with the fun characteristics. One of the most difﬁcult tasks is applying the serious information into the game while keeping the entertainment during the play. The balance between serious information and entertainment is a key of success. On the other hand, developing a serious game requires different experts (project manager, domain experts, storyboard writer, art director, pedagogical expert, programmers, etc.) working together. Unfortunately, there is no established development methodology to ensure effective coordination and integration across these disciplines, which makes the development process more difﬁcult. One of the key aspects of a serious game is realism. Rhyne et al. [181] suggests that the higher the level of reality or fantasy is, the more entertaining a game becomes. Realism, including visual, physical and functional, motivates players to accomplish the goals of the game and hence becomes fundamental to ensure the effectiveness of teaching/learning.

1.2 Objectives

The main goal of this thesis is the study and development of new techniques capable to improve three main aspects involved in the development of a serious game. The first one is the teaching/learning component of the game which relates with the capability of the game to transmit some predefined skills to the user. The second one is the graphic realism which influences user perception on the game. The last one is the user interaction which aims improving the physical realism while playing. Our investigation will be focused on serious games for health and medicine, con- 1.3. Thesis outline 9 sidering the CPR protocol as our key study. We aim at developing a new serious game to teach and learn CPR. We will denote this game LISSA which stands for Life Support Simulation Application. We will work together with the Faculty of Infirmary of the Uni- versity of Girona from Spain and the Faculty of Technology and Environment of Price of Songkla University from Thailand. To reach our objective we plan to:

Study and survey serious games for health and medicine that have been proposed • in the last decade. Due to the importance of this application area in the context of serious games we aim at investigating characteristics and state of the art of serious games for health. We will propose new classiﬁcations of these games with the idea to make its evaluation easier.

Study the CPR protocol and develop a serious game to teach and learn CPR. • All the game actions will turn around a CPR scenario that will reproduce an emergency situation that requires CPR procedures. The CPR scenario will be deﬁned by the instructor and presented to the learner as a test or problem. The learner will solve the problem applying the CPR procedures in a game mode.

Study the visual realism focusing on 3D serious games and their elements. We • will study and evaluate photo and non-photo realistic effects in the context of LISSA, our developed game. We will also consider the camera viewpoint by comparison 1st and 3rd person view.

Study the physical realism in order to improve the interaction between game and • player. Our idea is to propose new strategies capable to reproduce important factors of the CPR protocol such as compression rhythm, chest position, compression depth, hand position and breath giving.

Evaluate the proposed framework from teaching/learning point of view in a real • scenario focusing on CPR skill improvement for learner and usability for the instructor. This work will be carried out in collaboration with the Faculty of Inﬁrmary of the University of Girona from Spain and the Faculty of Technology and Environment of Price of Songkla University from Thailand.

1.3 Thesis outline

Besides this introductory chapter, this document is organized into the following chap- ters.

Chapter 2: Background • In this chapter, the background on serious games required for the comprehension of the main issues that are going to be analysed in this thesis is introduced. The core process of serious games and their functionalities will be described.

Chapter 3: Serious games for health • In this chapter, a survey of more than one hundred serious games for health is 10 Chapter 1. Introduction

presented. We will propose new four classiﬁcations to categorize all these games. We will analyze all the games with respect to these classiﬁcations.

Chapter 4: LISSA, a Life Support Simulation Application • In this chapter we will present the developed serious game to teach and learn CPR. We will review the state of the art on serious games and application to learn CPR. Then we will describe the main modules that compose our serious game.

Chapter 5: Visual realism design for LISSA • In this chapter, we will present the elements and the process of visual realism design for 3D serious games. The 3D object design of LISSA will be considered as a case study. Furthermore, an evaluation of visual realism with regard to photorealism and camera viewpoint will be presented.

Chapter 6: A Kinect-based system for LISSA • In this chapter, we will study different human motion tracking tools for CPR in order to select the proper interaction device for our game. We will propose a Kinect-based system for our serious game [139]. In addition, a comparison of three CPR feedback systems with regard to the chest compression rate and correct arm position will be presented.

Chapter 7: LISSA as an e-learning environment • In this chapter, we will evaluate the developed serious game from an e-learning point of view. We will assess on a real context the impact of LISSA with respect to CPR teaching strategies based on manikins.

Chapter 8: Conclusions • Finally, conclusions of the thesis and future work will be presented. In addition, the publications related with this thesis will be summarized. CHAPTER 2 Background

2.1 Introduction

SuperData Research group reported that the US digital games market grew 39% in January, 2013, compared to the same month last year with a totalled 960 million dollars [204]. Serious games are a part of the game market which is also growing fast and becoming more and more popular. A serious game is defined as a game with specific intention such as education, training, treatment, skill enhancement, widely used in many areas including military, health care, business management, or social science. Digital based serious games started appearing in the early 1970s for educational purpose [102]. Up to this date, significant improvements to real-time hardware and software techniques realized new possibilities for computer generated images, animations and graphics for games. In this chapter we will focus on serious games, describing their main functionalities, components and also the core processes involved in the development of a serious game. Beside, the review on serious games as an e-learning tool, 3D production for games, the three types of realism in games which are functional, visual and physical will be presented.

2.2 Serious games

A game is a physical or mental contest with a goal or objective, played according to a framework, or a set of rules, which determines what a player can and cannot do inside a game world. All games, including computer games, can be speciﬁed by means of four different functions [91] (see Figure 2.1).

The ﬁrst function is the rule or gameplay, which creates the pattern deﬁned • through the game rules that connects the player and the game.

The second function is the challenge, which determines the bonuses to reward • the good actions or the obstruction and barriers that avoid the player reaching the game goal easily. Challenges are used to create the different difﬁculty levels of the game in order to encourage enjoyment and motivate the player to spend more time with the game.

The third function is the interaction which represents the way the player com- • municates with the game. Interaction refers to any action that is done by the 12 Chapter 2. Background

Figure 2.1: The differences between games and serious games functionalities

player to start some activity. Interaction can be visual, listening, physical (typ- ing, mouse, touchpad, button pressing), dialogue exchange, etc. The different interaction techniques are supported by speciﬁc tools, which represent equipment or accessories that are connected to the games to give input information to the system. For example, the player may use a Wii balance board as a tool in an exercise game to improve his or her motor skills after operation.

The last function is the objective which is deﬁned as something that one’s efforts • or actions are intended to attain or accomplish. Two types of objectives can be distinguished: explicit and implicit. While the explicit objective is only entertainment, nature of every game, the implicit objective includes increasing skills and abilities, gaining knowledge or acquiring experience. The type of objectives can differentiate computer games (with only explicit objectives) from serious games (with include both implicit and explicit objectives).

2.3 Core components of a serious game

The creation of a serious game involves different processes, technologies and special- ists. Below we describe them and in Figure 2.2 we illustrate how they are related. 2.3. Core components of a serious game 13 Core components of serious games Figure 2.2: 14 Chapter 2. Background

A first component that can be considered in a serious game is the developer team which may consist of managers, 2D and 3D graphic designers, programmers, researchers and content providers, amongst others. In a small team one person may be assigned to more than one position; also in medium or big development teams other positions will be added [62] such as game designer, game tester, level designer, animator, software engineer, interface artist or simulation analyst. The number of people in the team depends on the budget, time, and size of the serious game. In the case of serious games, special attention must be paid to the content provider who provides not only the information related to the game but also shares his expertise to define game parameters such as the level of difficulty or the proper rewards and obstructions. Generally, content providers are professionals and experts in a specific application domain. To create the serious game, the developer team has to determine the tools, the technologies and the contents that have to be used in the game. Below we describe these elements.

Tools can be separated into three main groups that work together, the game en- • gine, the database and the design software application. First, all art assets in both 2D and 3D formats are created (by various design applications, such as Adobe illustrator, Coral Draw for 2D art assets and Autodesk 3DMax, Maya, Zbrush for 3D assets) and kept in the database. The database maintains all the data and information required by the platform including player information, score, game object, animation, etc. The game engine is one of the most important components and contains the specific code that controls how the system on each game operates. When an application is launched, the game engine dictates what the graphic user interface (GUI) will present to the user. It also defines the set of rules that determines the win conditions, i.e., the necessary steps that a user must take to complete the game. Every input information is gained through GUI; then, feedback action is generated by the game engine which is connected to the database. The engine receives the user inputs, interprets what the user is attempting to do and describes the proper outputs. When the win conditions are satisfied, the proper interface is presented and the game scores are recorded for future reference.

Content can be deﬁned as signiﬁcant information which will be delivered to the • players when the serious game is played. Content is provided by experts and converted to useful information according to the objective of the serious game.

Technology is the branch of knowledge that deals with the creation and the use of • technical means and their interrelation with life, society, and the environment; drawing upon such subjects as industrial arts, engineering, applied science, etc. In this context, two outstanding technologies are virtual and augmented reality [201][120]. Virtual reality refers to computer-simulated environments that can simulate physical presence in places located in the real world, as well as in imaginary worlds [246]. It is commonly associated with immersive technology which provides perceptually-real environments by special equipment such as 2.3. Core components of a serious game 15

holography, head-mounted displays (HMDs), haptic tactile equipment, etc. Aug- mented reality is a technology which allows computer generated virtual imagery to exactly overlay physical objects in real time [243].

These three components (tool, content and technology) are fundamental in the game development process. To select them we need to identify the game objective and the game genre, both described below.

Game objective, in the case of serious games, is mainly focused on education, • training and informing in an effective and incisive manner [80][99][183][241]. Other objectives can be practice, testing, simulation, diagnosis and treatment. A serious game can contain more than one objective. For example, serious games for health are not only focused on edutainment but they also show many good results on diagnostic and treatment as well.

Game genre is used to categorize video games based on their gameplay [23]. • Some of the proposed game genres are: adventure games which involve exploration of, and interaction with, the environment as a main facet of gameplay; Sport games which emulate traditional physical sports such as basketball, golf football, etc.; Strategy games where players require tactics and sagacity to achieve the targets; Exergame which combines exercise equipment with video games, to encourage people to exercise by making activity more fun; Puzzle games which require the player to solve a puzzle such as a maze, logical problem or positioning different pieces together; Action games in which players are required to have good reﬂexes, hand-eye coordination and quick reaction times in order to overcome challenges such as combats, avoiding traps, jumping, running, completing tasks within a pressing time limit, etc.; Role Playing Games in which the player’s character has skills and abilities represented by statistics. The gameplay involves the characters exploring and completing quests that build up their statistics and possessions. The game can be single or multi-player; and ﬁnally Simulation games that attempt to realistically mimic the conditions of a particular environment or activity.

The target player speciﬁcation is one of the more important parts of the game development process. The objectives are creating the most satisfactory game for the intended users and also obtaining the most efﬁcient results on implicit objectives, as mentioned in Section 2.2. The player can be characterized by various contexts such as playing style (socializer, killer, achiever and explorer), playing skill (newbie, casual and hardcore) and status of player (children, disabled and elderly). Focusing only on the player status will be not enough to encourage repeated use when the target player skills are upgraded. In order to satisfy players of all levels, the challenges need to be carefully considered providing suitable rewards and obstacles. A cross study of various combinations of player types would be interesting for serious games developer teams in order to satisfy all target players. 16 Chapter 2. Background

Game platform, also called a video game platform or video game system, refers • to the speciﬁc combination of electronic or computer hardware which, in conjunction with software, allows a video game to operate. Game Platform can be separated into four types: Arcade (Large devices normally found in commercial game centers); Console (NintendoNES, PlayStation3, Wii, Xbox360, etc.); Hand- helds (Game Boy, iPad, iPhone, Nintendo DS, PSP, Mobile, etc.) and Personal Computers (Desktop, Laptop).

Special equipments or interface devices used in virtual environments serve as por- • tals into a virtual world. The data input devices perform special interaction purposes such as gesture recognition or performance capture. For example, hand tracking, hand orientation, Wii-mote, pinch glove or motion tracking. On the other hand, the special devices for data output devices such as 3D vision goggles or 3D LCD screens are used for vision based image control on serious games.

As Figure 2.2 shows all described processes and elements must be considered together since they are related to each other.

2.4 Serious games as an e-learning tool

Serious game is a form of educational technology in teaching and learning. When a game designer focuses on learning outcomes while preserving playfulness, serious learning is possible [9]. Serious games can be applied to every level of education, in and out of school, for laypeople to professional, from kids to elder people and also can be applied to a wide variety of areas.

Figure 2.3: Dale cone of learning [41], which shows the degree or levels of learning that occurs when learners engage to each learning style (reading, hearing, seeing, watching, participating and doing) 2.4. Serious games as an e-learning tool 17

The potentiality of serious games as a tool for learning has been acknowledged in the ability to balance the entertainment, interactivity and re-playability of typical games with the learning goals of a given educational objective. Moreover, serious games approach learning as a challenging, difficult and rewarding in order to increase players engagement. An increased effectiveness of learning with serious games has been generally accepted. For example, lower time is taken to become productive and effective on the domain topic [174]. In addition, with regard to Dale cone of learning [41] (see Figure 2.3) which shows the degree or levels of learning that occurs when learners engage to each learning style (reading, hearing, seeing, watching, participating and doing). The learning outcomes are increasing from top to bottom (from reading to doing). Therefore, learning by simulating a real experience, which is the bottom level from the cone, could support the comprehension of what they are learning more than reading or hearing which are on the top level of the cone. Dale states that users could obtain 90 percent of what they participate in simulation. So, in order to create an e-learning tool with high learning efficiency and fun, in addition with the risk and budget awareness in real practice, simulation with serious games is the most interesting one. Focus on learning context, Pereira, G. et al. [164] proposed that serious game can support four learning categories which are; personal skills, e.g. strategic thinking and creativity; interpersonal skills, e.g. leadership, conflict management and team building; applied ethics, e.g. clinical and professional ethics; and social awareness, e.g. environment, violence and economic behaviors.

Figure 2.4: Example of serious games for learning

Figure 2.4 shows some examples of serious games from several areas. From top to bottom and left to right: The Virtual Dental Implant Training Simulation, a serious 18 Chapter 2. Background game designed to help dental students in diagnostics, decision making and treatment protocols [134] (see Figure 2.4(a)). Quest for oil, a serious games for oil exploration hands-on experience. It helps player learning about the layers of the earth, how complex it is to get the oil and what it takes to be in the highly competitive oil business [128] (see Figure 2.4(b)). Cruise ship evacuation sim, a serious game that aims at helping cruise ship crews train for several different disasters [59] (see Figure 2.4(c)). RescueSim, a training software that prepares public safety and security professionals for real-life incidents in a virtual environment [230] (see Figure 2.4(d)). 3D Networks, a serious game aim at training civil engineering students about the risks of public works near underground networks [14] (see Figure 2.4(e)). NanoMission, an educational serious game that aims to teach players about the concepts of nanoscience through real world practical applications from microelectronics to drug delivery [166] (see Figure 2.4(f)).

2.5 3D Production for games

One of the key aspects of serious games is the production of the scenarios where the game will be carried out. In this context 3D graphics play a decisive role. There are different techniques to create 3D graphics, such as 3D sculpture, 3D scanning and image-based 3D modelling. In the context of 3D graphics for games, image-based 3D modelling is the most common used technique. Generally, the technique uses a low-polygon modelling strategy in order to reduce ﬁle sizes and enhance game performance. The traditional 3D production process can be divided as the following ﬁve basic steps:

The ﬁrst step is modelling, the process of forming the shape of an object. The two • most common sources of 3D models originate from 3D modelling tools and 3D scans of real-world objects. Basically, a 3D model is formed from points called vertices that deﬁne the shape and form polygons. A polygon is an area formed from at least three vertices (a triangle). The overall integrity of the model and its suitability to use in animations depend on the structure of the polygons. It has to be taken into account that games are real time rendering systems that produce dozens of frames in each second. Thus, 3D models particularly suited for games must be low poly models, i.e., they must contain a reasonably small number of polygons. The creation of low poly models is an important factor to optimize game performance, however these models have to preserve the appearance of the graphics. Therefore, a trade-off between number of polygons and visual realism of the object is required [42]. The number of polygons and objects which can appear in a scene and can still be rendered at an acceptable performance, has to be carefully calculated and tested in order to maintain the smoothness while playing. In Figure 2.5 we illustrate an example of the polygons used to create a 3D scene. 2.5. 3D Production for games 19

Figure 2.5: Polygons used to model a hospital scene

The second step of the 3D production process is texturing or texture mapping, a • method that maps the texture of surfaces to the 3D geometry produced in the modelling step. Every vertex in a polygon is assigned a texture coordinate either via explicit assignment or by procedural deﬁnition. Multi-texturing is the use of more than one texture at a time on a single polygon. For instance, a light map texture may be used to light a surface to avoid lighting computation each time the surface is rendered. Another multi-texture technique is bump mapping, which allows a texture to directly control the facing or normal direction of a surface for the purposes of its lighting calculations; it can give a very good appearance of a complex surface, such as tree bark or rough concrete, that takes on lighting detail in addition to the usual detailed colouring. These computer graphics techniques have been designed to visualize low poly objects as well as the high poly ones. Figure 2.6 shows the scene presented in Figure 5.1 with textures mapped onto the polygons.

Figure 2.6: The hospital scene with textured polygons

The third step of the 3D production process is lighting or illumination, which is • usually carried out by a game engine. Different lighting techniques can be used to obtain a realistic rendering. There are also different types of light sources providing customization for the shading of objects. Amongst them, ambient lighting, directional lighting, point lighting, spotlight lighting, area lighting and volumet- ric lighting. An ideal lighting environment requires detailed descriptions of the surfaces to allow lighting recalculations at runtime [198]. Since the lighting calculation for games is processor-intensive, some lighting effects such as area 20 Chapter 2. Background

lights are not available at runtime and are solved by using light-maps during the texturing step. More advanced lighting effects such as shadows and reﬂections can greatly increase the realism of games. However, real-time computation of these effects often requires high-end graphics hardware. Figure 2.7 presents the hospital scene after lighting applied.

Figure 2.7: The hospital scene after lighting applied

Figure 2.8: A same scene illuminated with different light sources

The fourth step of the 3D production process is animation which refers to the • temporal description of an object, i.e., how it moves and deforms over time. Popular methods include key-framing, inverse kinematics and motion capture, although these techniques are often used in combination. As with modelling, physical simulation also speciﬁes motion. Animation techniques for games are as similar as for other media, the game engine also facilitates some turnkey solution for animations such as animation re-targeting, data-driven controllers (with motion capture system) etc.

The last step of the 3D production process is rendering which converts the 3D • virtual world model into a 2D image (a sequence of images), which can be photorealistic (see Figure 2.9) or non-photorealistic rendering (see Figure 2.10) depending on the applied effects. In the game context, the rendering step has to be performed in real-time. 2.5. 3D Production for games 21

Figure 2.9: A realistic rendering of the hospital scene

Figure 2.10: The hospital scene rendered using different not-photorealistic styles 22 Chapter 2. Background

2.6 Realism in games

In digital art, realism is an aesthetic concept that promotes accurate, detailed, unem- bellished depiction of nature or objects, rather than an imagination. The immersive experience in 3D environments can be improved by increasing the degree of realism [188]. The realism in games can be separated into three types: physical, functional and visual realism [69, 129], as illustrated in Figure 2.11.

Figure 2.11: Type of realism in games, with regard to [69] and [129]

2.6.1 Functional realism

Functional realism is deﬁned by the actions through user interface by graphic user interface (GUI) widgets, which are pieces of objects that display an information navi- gated by user. For example, in the golf game to hit the ball user has to control different parameters, the type and number of the golf club, the hit direction, the club head speed, and the velocity at which a golf club impacts with a golf ball. Figure 2.12 shows the club head speed function of three different golf games. All three games have a similar idea to present this action, player need to hold the mouse click and release when the pointer arrives to the intended scale.

Figure 2.12: Club head speed function of three different golf games 2.6. Realism in games 23

2.6.2 Visual realism

Visual realism is determined by the degree of accuracy at which real objects are represented. This depends on light interaction, virtual objects and animations. There are five steps to create a 3D game objects, modelling, texturing, lighting, animation and rendering. Each step can be applied variously from abstract to the real, more details are presented in Chapter5. Realism is a user dependant definition since the degree of satisfaction with respect to realism depends on the user requirements. To illustrate this we present in Figure 2.13 screen-shots of three different soccer games which have different levels of realism. Note that the left one (Striker Soccer Euro 2012) has less realism than other two mainly on the characters style, which has not a real proportion. The characters style of the other two games are quite similar although the right one (Pro evolution soccer 2013) is more outstanding in realism according to the cloths, color of the field and lighting. For people who are not familiar with games, all the games might be categorized as 3D games and no differences between the FIFA soccer Euro 2012 and Pro evolution soccer 2013 will be detected. On the contrary, an expert will be able to detect all the differences between them. Therefore, to create the art assets for games the authors need to define the level of realism that optimizes user satisfaction, specification of the equipments and resource usage.

Figure 2.13: Screen-shots of three soccer games which have different level of realism, increasing from left to right

In order to optimize the visual realism for a serious game, player perception is the most important issue that need to be considered. The successful serious games may not be very similar to the real world but should be real enough to motivate players to play the game until they achieve the game objectives and get the satisfying learning outcomes. Many researchers concur that player perception can be inﬂuenced by visual realism [39, 188]. Visual realism is distinguished by the term Visual Fidelity which refers to the degree to which visual features in the virtual environment conform to visual features in the real environment [26, 104]. In order to convince the player that the virtual environment is real, the scene presented should faithfully model the expected actual environment. A highly accurate, fully modelled, interactive environment is thus seen as virtually real. 24 Chapter 2. Background

Visual realism has ﬁve basic building blocks: form (or geometry), texture, illumination, movement (including physics) and viewpoint, as illustrated in Figure 2.14.

Figure 2.14: Visual realism components

Form is created by points, lines, curves, or surfaces in the modelling step. To • model an object for games, the form can have real proportions or distorted depending on the game concept.

Figure 2.15: Different character proportion

Texture of the 3D model can be separated in two types, bitmap (or raster) based • texture and vector based texture. In computer graphic, bitmap based texture refers to the mapping of color on pixels, where each one may store more than two colors, thus using more than one bit per pixel, for example photograph, scanned picture, etc. Vector based texture uses of geometrical primitives such as points, lines, curves, and shapes or polygons to represent an image. Generally, bitmap presents better realism than vector based texture. Figure 2.16 presents vector and bitmap-based grass textures. Note that bitmap-based looks more realistic than the vector-based one (see Figure 2.16(a) and (b).

Illumination or light can be applied into the games in two ways. The ﬁrst one • is placing the light sources directly into the scene, in this way the results can be created as fantastic as artist can imagine. Figure 2.17 shows the comparison of fantastic light (Hunted: The Demon’s Forge) and realistic light (Call of Duty). The second one is applying some lighting effect onto the textures, known as Image-Based Lighting, which saves a lot of light processing resources while playing but requires more creation time. Image-based lighting is the concept 2.6. Realism in games 25

Figure 2.16: Comparison of vector-based and bitmap-based textures

Figure 2.17: Comparison of fantastic light (Hunted: The Demon’s Forge) and realistic light (Call of Duty)

to place some pre-deﬁned light effect onto the textures. There are three light characteristics, which are absorption, reﬂection and refraction, that use to create image-based lighting for video games. These characteristics can be applied to several light maps such as diffuse map, specular map, ambient Occlusion map, etc.

Movement in games is related mainly to two things, objects and camera. Focus • on realism, the movement of both 3D objects and camera are commonly required to be as much natural as they can. In order to do that some tools such as motion tracking devices or techniques such as rotoscoping, an animation technique that animates frame by frame with the guide of real movement in video [239], are required.

Viewpoint refers to a visual perspective rendered from the pre-deﬁned camera • position. There are a wide variety of viewpoints that players can go along within games world by the setting of: camera shot, amount of space that is seen in one frame such as closed-up shot, medium shot and long shot; camera angle,

position the viewer such as bird0s eye view, over shoulder and worm0s eye view; and camera viewpoint, view that user sees the action through the eyes of the game character such as 1st and 3rd person view. The viewpoint and perspective roles are the least studied elements of realism. Rob Pepperell discusses in [163] the problem of how to depict the relative indis- tinctness of peripheral vision as compared with central vision, and the appear- 26 Chapter 2. Background

ance of our bodies in our ﬁeld of view. He comments as some famous artists like Paul Cézanne and Vincent Van Gogh present their works as observers are outside looking in. This means ﬁrst person view, which intended to simulate viewing the scene through our own eyes, may be less realistic due to the lack of ability to perceive our body, comparing to other, outer viewpoints.

Some researchers support that realistic images, known as photorealistic rendering (PR) images, often contain too many details disrupting the information to be delivered [121]. Thus, artists often simplify images highlighting information using image stylization techniques. The obtained images are known as non-photorealistic rendering (NPR) images. The creation of these images has lead to a new area of investigation in graphics known as expressive rendering [115]. Figure 2.18 shows NPR effects applied to a game scene, from left to right we have applied edge enhancement, texture simpli- ﬁcation, shadow recoloring and depth sensation. For games, these effects will applied as a post processing step during real-time rendering.

Figure 2.18: Examples of non-photorealistic rendering (NPR) applied to a game scene

In Figure 2.19 [131], we show a same scene rendered using different surface formats (bitmap and vector graphics) as well as different rendering effects.

Figure 2.19: The ten render styles ranking from abstract to realistic [131]

2.6.3 Physical realism

Physical realism for games is determined by the use of any peripherals or interaction tools that persuades players to do or feel like in real situation. Focus on games, physical 2.6. Realism in games 27 realism can be enhanced by using two categories of interactive device which are motion tracking devices and haptic devices,

Motion tracking device, also known as motion capture, is a motion sensing in- • put device aimed at collecting the set of sequenced 3D position on object or human. Human motion tracking devices can be separated to many different aspect such as: wired/wireless; optical/non-optical based; realtime/non-realtime; and marker/marker-free. The price varies depending on accuracy, numbers of cameras, etc.

The tracking devices for games are commonly created for human rather than for objects, real-time rather than non-realtime and market less rather than marker based. Two motion tracking leaders on global games market are Kinect [139] and Wii [154]. Some examples are: ﬁshing cactus, a Kinect based serious game for patients who has a lack of logic and organizational skills [74] and a collection of wii based therapeutic games for parkinson disease patients to increase their balance [177];

Haptic device is a tactile feedback equipment that takes advantage of the sense of • touch by applying forces, vibrations, or motions of the user. In some touch-based system such as surgical simulation, the haptic device is not only made to improve the physical realism, but also to provide important diagnostic information through the sense of touch [242]. Figure 2.20 shows an example of a training simulation that enhance the physical realism by using Novint technologies in order to train and utilize the sense of touch of the players [158].

Figure 2.20: VRDTS (Virtual Reality Dental Training System), the dental training simulator that utilizes a 3 dimensional haptic interface to enhance the physical realism while playing [158]

The lastest generation of haptic devices (4th generation) is focused on pressure sensitivity, enabling how hard you press on a ﬂat surface to affect the response [237]. 28 Chapter 2. Background

2.7 Conclusion

Serious games are games used for purposes other than mere entertainment. They inherit gameplay characteristics from entertainment games and have focused on the main objective such as learning or training, with the idea to apply learnt lessons in real- life work environments. In this chapter, we have presented the core process of serious games and their functionalities. In addition, we have reviewed serious games with relation to a learning model (Dale’s cone of learning) and some examples of learning serious games in different areas have been stated. We have also described realism in games focusing on the three types of realism: functional, visual and physical. CHAPTER 3 Serious games for health

3.1 Introduction

Our purpose is to study serious games for health. To reach this objective we have to determine what are considered as games for health. The World Health Organization (WHO) [238] defined health in its broader sense as a state of complete physical, mental, and social well-being and not merely the absence of disease or infirmity. Other definitions simply require being free from illness or injury [231]. If we use the stricter definition, we may only consider games dealing with the different phases of illness development: both doctor training and patient familiarization with his illness. However, the use of the WHO definition allows us to consider a third variety of games which has recently had a big success. These are games dealing with healthy habits such as exercise (including dancing and fitness games). We will use this broader definition to select the games. The purpose of this chapter is to survey serious games for health, present new classifications regarding their different aspects and analyse each game based on the functionalities described in the classifications.

3.2 The use of serious games to promote health

In the last decade, many serious games in the field of e-health have been developed [185]. These deal with a wide variety of aspects such as surgeon training, radiology operation, cardiopulmonary resuscitation (CPR) and patient care. Different surveys on serious games for health have been published. Watters et al. [235] explored the use of games for children with long-term treatment regimes, where motivation for compliance is a key factor in the success of the treatment. Papastergiou [161] reviewed thirty-four articles on the use of computer and video games in health education and physical education and presented a synthesis of the available empirical evidence on the educational effectiveness of them. Kato [105] summarized the scientific literature of commercially available and tailor-made games used for education and training with patients and medical students and doctors; her classification is based on diseases. Rego et al. [179] proposed a classification designed to properly distinguish and compare eight serious games for rehabilitation systems with respect to their fundamental characteristics. They also described a particular serious game for rehabilitation, Reha- Com, as a case study. Lopes and Bidarra [126] presented the state of adaptability in general games and simulations focusing on the purposes, targets and methods from both academia and industry. Bartolome et al. [15] presented a systematic review of 30 Chapter 3. Serious games for health twenty-one serious games for health and education, described at scientific papers, and projects from the 7th Framework Program. The main novelties of our survey with respect to reported ones, are the new classifications of serious games focused on health and the number of considered games (one hundred and nine from January 2004 to December 2012).

3.3 Method and scope of study

In this survey we have considered not only serious games that have been described and evaluated in peer-reviewed publications but also commercial games (consoles and PCs), online games, games on mobile platforms and games running on specialized platforms in clinics, hospitals and patients homes. To carry out our research, we have considered the following sources: the international online bibliographic databases of Science Direct, Association for Computing Machinery (ACM), Institute of Electrical and Electronics Engineers (IEEE) Computer Society Digital Library (CSDL), Cambridge Journals Online, Oxford University Press (journals), NRC Research Press, The CINAHL database, BioMed Central and Emerald. We have found other references using Google and Google scholar. Our searching key words have been game, video games, play, serious games, simulation, virtual, reality, game based learning, training, health, clinical, treatment, rehabilitation and medicine. Some speciﬁc keywords may have been derived from a previous search: such as role play game or RPG, exergame and edutainment. All the terms have been addressed in conjunction to increase the efﬁciency of possible outcomes.

Figure 3.1: Numbers of surveyed serious games for health according to the year of publication

If we focus on the number of serious games for health according to the year of publication from January 1989 to December 2012 which we present in Figure 3.1, we can see that before 2004 few games were proposed while from 2004 until now the production increases. Therefore we decided to take into consideration only the serious games for health published from January 2004 to December 2012. To be included in this review, papers must show empirical evidence related to the main objective of this 3.4. Review of serious games for health 31 thesis (serious games for health and their many aspects).

3.4 Review of serious games for health

In this section, we present the games that have been considered for our study. In Table 3.1, we review the work developed in the health area, including additional games from commercial or online sources. We provide a short summary of each game, indicating their most relevant characteristic by year of publication.

Year Developer/Author Game name Description 2004 Johnston, E and Ben Game A serious game designed to help kids fight Duskin, B against cancer. The game was initiated by Ben Duskin, nine years old, who is in remission from Leukemia [60]. 2004 MIT Teacher Educa- Outbreak @ MIT A handheld game occurring within the tion Program premises of MIT that aims at exploring an epi- demic of avian influenza being triggered in college [145] 2004 Montreal Science Cen- Sleep: A to Z A serious game which aims to present the op- tre eration and the importance of the different stages of sleep by a set of mini games [146]. 2004 Respondesign MayaFit An exergame dedicated to physical training. Many fitness exercises are offered by virtual coach. [180]. 2005 Nintendo Brain Age 2 A serious game that evaluate the brain age, between 20 to 80, by determining approximately the range of brain responsiveness. [152]. 2006 Archimage, Inc. Escape from Diab An adventure game in healthy eating and exercise that focuses on obesity and type 2 diabetes prevention. The game guides the player through goal setting, problem solving, energy balance, and other game-play activities [11]. 2006 Believe in Tomorrow FreeDive A serious game that helps chronically ill chil- National Children dren cope with pain and anxiety by distract- Foundation, Ltd. ing them while they undergo painful medical treatments [18]. 2006 Gameloft Brain Challenge A collection of serious games aiming to exercise the brain and keep intellectual muscles in shape. It also checks the performance and monitors the overall results of the brain activity with the daily follow-up. The game is based on four subjects: logic, math, memory and visual [79]. 2006 Janomedia Terveellinen Ateria A serious game aiding practical nurses (PN) and school staff train in the practical aspects of preparing meals for people with different nutritional requirements [100].

Table 3.1: Review of serious games 32 Chapter 3. Serious games for health

Year Developer/Author Game name Description 2006 McGIll University Grow Your Chi A serious game designed to increase self- esteem by displaying a positive ability or function of the player [132]. The game asks the player to identify a smiling face and the player name from many photographs of frowning faces and lists of other names. This creates the ability to identify a smile or a happy face from plain, angry or sad faces. 2007 BreakAway, Ltd. Pulse!! The vir- An immersive virtual learning space where tual clinical learn- health care professionals can train their clini- ing lab cal skills by dealing with injured patients, bio- terrorism or other catastrophes [5, 25]. 2007 Edheads Virtual Knee Re- A science game for patients and their care- placement Surgery givers to take on the role of a surgeon and complete a knee replacement surgery while learning about the procedure, the technology, and health risks and beneﬁts [54]. 2007 Fatworld.org Fatworld A video game that explores the relationships between nutrition, obesity and socio- economic factors in the contemporary U.S. Budgets, subsidies, regulations and physical world characteristics are taken into account [67]. 2007 Intelligent System Co., Otona no DS Kao A brand game released only in Japan. The Ltd. Training training software instructs the user on how to perform several facial exercises that have a goal of keeping your face healthy looking and wrinkle free [97]. 2007 Nintendo EAD WiiFit An exercise game consisting of activities utilizing the Wii Balance Board peripheral. Wii Fit Plus, an enhanced version of the original Wii Fit, was launched in 2009 by adding new exercises and tools to personalize the exercise routine [153, 154]. 2007 Nordic Innovation Valion Energia- A nutrition game aimed to build a healthy and Centre summaaja balanced meal. The effects of the meal on blood sugar are shown, and possible improvements in the meal are suggested [31]. MC Urho A game that contains information regarding lifestyle effects on health and aims to teach young people about the effects of smoking, high blood pressure and cholesterol [155]. MoFun Circus An exergame in which players cycle to capture falling objects in the game. A camera is used to follow the users and display the activity on-screen [156]. Table 3.1 Review of serious games (continued) 3.4. Review of serious games for health 33

Year Developer/Author Game name Description 2007 SEGA Corporation Mind Quiz Series of 16 mini-games for mental training, (Nounenrei) aiming to measure and improve particular parts of the player brain, such as one brain age and its brain stress degree [194]. 2008 Anderson C. - A simulation game was created to help nursing students to identify the roles and respon- sibilities of a multidisciplinary team of professionals when caring for a Maternal-Child Health (MCH) patients and family in crisis [8]. 2008 Glasgow Caledonian Nurse Education A virtual learning environment in Second University Life, a massively multiplayer online role playing game (MMORPG), for use in nurse education [82]. 2008 Hatﬁeld D. My Stop Smoking An educational game for smokers to quit im- Coach mediately and permanently. The game is run on several platforms including iPhone, Mobile (Java ME) and Nintendo DS [86]. 2008 Imperial College Game-based A region in Second Life provides a learn- London, Faculty of Learning for ing space where virtual patients suffering ﬁve Medicine Virtual Patients different respiratory illnesses (such as lung cancer or pneumonia) can be diagnosed, investigated and treated by players wanting to perform role-playing learning activities under the feedback and guidance of medical staff [66]. 2008 Mili et al. VI-MED A virtual training to be used as a precursor and as a supplement to real practical training [141]. 2008 Sliney and Murphy Medical Simu- A computer-aided junior doctor simulator lation Training used for training and teaching junior doctors Program (JDoc) their interpersonal, communication and decision making skills, and to ease the transfer- ence of the medical information available to them [200]. 2008 The Partnership for The Food Detec- A web based game for 8-12 year old kids to Food Safety Education tives Fight BAC learn about foodborne illness [211]. game 2008 TruSim Triage Trainer A serious game to train in triage, the process of prioritising the treatment of multiple casu- alties based on the severity of their injuries. 2008 Vermont department Khemia A serious game designed to help people quit of health smoking. The game provides both a distrac- tion from cigarette cravings and personalized support for quitting through the integrated MyQuitKit tool [224]. 2008 Warner Bros. Enter- Pamoja Mtaani A video game that simulates real life set- tainment, Inc. tings in Kenya capital Nairobi aims to teaching Kenyan youth how to avoid contracting HIV [234]. Table 3.1 Review of serious games (continued) 34 Chapter 3. Serious games for health

Year Developer/Author Game name Description 2009 BBG Entertainment Train Your A brain-exercising game with 30 specifically GmbH Brain With Dr. designed and scientifically tested exercises. Kawashima Along with the goal of the game, it explains how the brain will be activated and developed by the training [17]. 2009 Blitz Games Studios, The Biggest Loser A health and lifestyle game. Based on the hit Ltd. NBC reality TV show, USA. The game mirrors the format of the show by featuring intense training routines, weekly challenges, nutritional goals and information and the iconic weekly weigh-in and elimination from the show [20]. 2009 Burke et al. Catch task A serious game for upper limb stroke rehabilitation (focused on bilateral rehabilitation) [27]. Whack a Mouse A serious game designed to encourage movement and to improve the accuracy and speed of the user upper limb movement [27]. Rabbit Chase A serious game developed for single arm rehabilitation (either right or left arm) [27]. Arrow Attack A serious game developed for bimanual rehabilitation (both arms) [27]. Virtual Vibraphone A serious game that uses Nintendo Wii remote controllers for wrist and arm rehabilitation [27]. 2009 Collision Studios Daisy Fuentes A fitness game of pilates exercise, a system of exercise created in the 1920s by Joseph H., features a 3D Avatar of Daisy Fuentes who performs the exercises with the player [38]. 2009 Deponti et al. DroidGlove A ubiquitous game therapy for wrist rehabilitation. The exercise has to be done while holding the smart-phone in the hand. The performance will be automatically recorded for the doctor supervision [47]. 2009 Hopelab Re-Mission A video game with 20 levels that takes the player on a journey through the body of young patients with different kinds of cancer. The main aim is to engage young cancer patients through entertaining game play while impacting specific psychological and behavioral outcomes associated with successful cancer treatment [206]. 2009 Keele University - A system which aims to train pharmacists by (2009) using a virtual patient. Traits such as race, age and gender are taken into account in the treatment of patients to let learners understand the clinical significance [193]. Table 3.1 Review of serious games (continued) 3.4. Review of serious games for health 35

Year Developer/Author Game Name Description 2009 Kim et al. A Sensory Gate- PC-based 3D graphics game designed for Ball Game aged people; it uses a realistic gate-ball stick and balls as interfaces. In the game, players use the same stick and ball as the real gate- ball [106]. 2009 Laikari A. Fitness Adventure A location-aware fitness game which takes advantage of a variety of mobile phones, location information and Bluetooth GPS re- ceivers; combines mobile games with exercising outdoors [113]. 2009 Learning Games Lab Science Pirates: An adventure game allowing the child to The Curse of learn about food safety and the underlying Brownbeard scientific principles. The adventure is made up of different challenges: problem solving, scavenger hunt, etc [116]. 2009 Lightning Fish Games NewU Fitness First A fitness game featuring both structured ex- Personal Trainer ercise programs and nutrition programs. The fitness programs are designed by the Fitness First gym chain, the nutrition programs are in association with the You Are What You Eat television series [122]. 2009 Mckanna et al. 21 Tally A collection of 2D games used to detect divided attention unobtrusively, by using performance on a computer game designed to force players to attend to different dimensions of attention simultaneously in order to succeed [133]. 2009 Persuasive Games LLC Killer Flu A game about seasonal and pandemic flu attempting to explain how flu really mutates and spreads, and how challenging it can be for a deadly strain to affect a large population geographically. The player takes the role of the flu itself, trying to mutate and then spread in a variety of conditions [165]. 2009 QOVEO Prevenir la gripe A A serious game to raise public awareness H1N1 about H1N1 virus prevention. It is a shooting game where player have to destroy viruses. However, in order to reload the weapon players have to answer questions related to the virus [171]. 2009 RANJ Serious Games The Great Flu A serious game to raise the awareness of similar outbreaks by having the player control the deadly Gamers Flu. The goal of the game is to control a possible pandemic by selecting options to apply actions or assign research teams in order to stop the flu [173]. Table 3.1 Review of serious games (continued) 36 Chapter 3. Serious games for health

Year Developer/Author Game Name Description 2009 Raylight S.r.l. Train Your Senses A serious game to train the visual and au- ral senses through 22 exercises. The game lets the player try first on his weaknesses and improve his skills through targeted training [175]. 2009 Succubus Interactive Happy Night Club A serious game which aims to sensitize teenagers about the risk of over-alcohol and binge drinking. Player performs as a secret agent sent on a mission in a night club [203]. 2009 Tost et al. PREVIRNEC A cognitive tele-rehabilitation system based on virtual environments. The system allows personalized treatments by means of 2D and 3D exercises that can be built according to single patient’s characteristics [213]. 2009 Virtual Heroes, Inc. Zero Hour: Ameri- A 4.8 million dollar serious game which aims and George Washing- can Medic to train and exercise first responders to re- ton University office of spond to mass casualty incidents such as homeland security earthquakes and terrorist attacks [227]. 2009 Vtnen and Leikas Virku - Virtual Fit- A system that allows users to exercise in a vir- ness Centre tual environment. The game is controlled by a user interface based on an exercise cycle, and users may practice individually or in a group [219]. 2010 Anchor Bay Entertain- 10 Minute Solu- An exergame which allows players to con- ment tion struct their own workout regimens based on 10 minute exercise sessions based on cardio boxing, mixed games, and aerobics [7]. 2010 Atkinson and - A medical diagnostic gaming environment Narasimhan that is used to gather parkinson patients information in a casual environment. The system employs the novint falcon human interface device [159] to guide a patient within the game [13]. 2010 Bartolome et al. - A serious game to analyse the behavior and promote certain social skills (conversation, negotiation, etc.) of people with Neurologi- cal development Disabilities [16]. 2010 BreakAway, Ltd. Virtual Dental A 3D virtual environment for students to train Implant Train- in the correct decision-making protocol to ing Simulation determine patients preparation (both physi- Program cal and mental) for dental implant surgery [25, 134]. 2010 Clawson et al. DITS A mobile phone game similar to Dance Dance Revolution (DDR). Instead of using a dance pad, DITS uses wireless 3-axis accelerometers

that are worn around the player0s ankles and uses a mobile phone to control the game [36]. Table 3.1 Review of serious games (continued) 3.4. Review of serious games for health 37

Year Developer/Author Game Name Description 2010 Electronic Arts, Inc. EA Sports Active An exergame focused on cardio exercise. Players can choose from three week or nine week programs which are rigidly guided systems that track players through the range of weeks selected [56]. 2010 Finkelstein et al. Astrojumper A stereoscopic virtual reality exergame for children with autism. During the game, virtual space-themed objects ﬂy forward toward the user who must use their own physical movements to avoid collisions [71]. 2010 Fishing Cactus R.O.G.E.R A serious game for patients who has a lack of logic and organizational skills (typically post- stroke patients, Alzheimer, hemi-negligent patients, etc.) [74]. 2010 Gago et al. Nutri-Trainer A collection of serious games about nutritional health following professional recom- mendations, cooperating with doctors and nutritionists to give coherence to the information collected in the nutritional databases [78]. 2010 Grau et al. - A neuropsychological rehabilitation game that allows patients to navigate through the virtual environment and perform cognitive tasks [85]. 2010 HopeLab Zamzee An online rewards program and game-like experience powered by your physical activity. Players wear a device with an accelerometer that monitors their movement and translates it to points that can be then redeems with both digital and real-world rewards [88]. 2010 Innovation in Learn- CliniSpace A medical training game for healthcare pro- ing, Inc. fessionals focusing on clinical diagnosis and patients management. Players in the role of a doctor who may consult medical records to a patient, plan an operation or provide a clinical consultation [95]. 2010 KTM Advance AlphegaGame An education game to train pharmacists focus on patients observation by counseling with the virtual patients in game [109]. 2010 Miller J. Market Virtual Pa- Patents simulations for students and pro- tient Care Simula- fessionals (nurses, pharmacists, paramedics, tion (MUVE) emergency medical technicians, social work- ers, etc.) training [142]. 2010 Sabri et al. - A serious game designed to train orthopedic surgical procedures to orthopedic surgical residents [186]. Table 3.1 Review of serious games (continued) 38 Chapter 3. Serious games for health

Year Developer/Author Game Name Description 2010 Skills2Learn, Ltd. Nursing and Mid- A game that helps nurses and midwives in- wifery crease their ability to assess patients. The interactive scenario is based on the simulation of the 36 weeks of pregnancy realistically [199]. 2010 TruSim Patient Rescue A serious game which supports health professionals to recognise the signs of patients deterioration, use set protocols to assess a patient’s condition and intervene effectively [214]. 2010 Ubisoft Divertisse- Your Shape: Fit- This console game focuses entirely on fitness ments, Inc. ness Evolved routines led by virtual trainers, and is divided into structured personal training, pick-up fitness classes, and active gym games [215]. 2010 Vidani et al. Emergency medi- A serious game for training in emergency cal services on vir- medical procedures concerning disabled pa- tual environment tients. It allows users to experience emer- (EMSAVE) gency situations involving disabled persons [87, 226]. 2010 Visual Imagination Chirurgie Simula- A surgery simulation in which players take on Software tor the role of a surgeon at a hospital. The surgical procedures include operating on fractures, removing an inflamed appendix or tonsils, treating infected gall bladders, attending to varicose veins, repairing hernias, restoring vision in cataract procedures and dealing with the injuries of a road traffic accident [228]. 2010 Wang et al. Brain Chi and The EEG-based concentration games named Dancing Robot Brain Chi (2D) and Dancing Robot (3D) are developed for concentration level control [233]. 2010 Zumba Fitness LLC Zumba Fitness An interactive exercise program helps to perform high-calorie-burning workouts. The game concept is based on calorie-burning dance fitness-party. Zumba Fitness is available in PlayStation 3, Wii and Xbox 360 platform [245]. 2011 Association RMC-BFM Staying alive A 3D simulation aiming to teach how to deal with emergency situation of cardiac arrest [12]. 2011 Botella et al. - A mobile phone game for the treatment of cockroach phobia. The objective is to reduce the level of fear and avoidance [22]. 2011 CCCP LudoMedic An educational game aims to teach children to prepare for an MRI, have surgery, or undergo chemotherapy [30]. Table 3.1 Review of serious games (continued) 3.4. Review of serious games for health 39

Year Developer/Author Game Name Description 2011 De Bortoli and Gaggi PlayWithEyes This serious game aims to test children eyes while they are having fun playing with Lea symbols and images taken from popular car- toons, using a touch interface [45]. 2011 Diehl et al. - A serious game for training doctors and medical students about insulin management for the treatment of diabetes mellitus. [51]. 2011 Fuchslocher et al. Balance A health game developed to optimize the self- management of teenagers with diabetes mellitus typeI [76]. 2011 Imbeault et al. - A serious game created specifically for patients suffering from alzheimer; advances in the field of artificial intelligence such as activity recognition and guidance to offer opti- mal experience through the training sessions [92]. 2011 Lakeside Center for Kinetix Academy A series of several games for autism by us- Autisme et Microsoft ing Kinect in order to stimulate: motor development, use of language and comprehension, use of different cognitive processes and social interactions [114]. 2011 Lin JK. - An augmented reality serious game for facil- itating the patients to execute rehabilitative activities without any geographical or time limitations [123]. 2011 Moya et al. - A 3D virtual environment for neuro- rehabilitation of the upper limb. Patient wears a special suit with sensors integrated to move one of their arms trying to simulate concrete daily actions, such as grasping a bottle, opening a door or putting a book on a shelf [147]. 2011 Nauta and Spil - A educational diabetes game which aims to enhance a healthy lifestyle by educating and coaching self-monitoring, reinforcements and observational learning [149]. 2011 Public Health Agency Buffet busters A game developed for grade 5 educators and of Canada students to promote infectious disease awareness and to introduce concepts related to food and waterborne infectious diseases as well as basic principles of epidemiology [169]. 2011 Queiros et al. - A low-cost laparoscopy simulator, for novice surgeons training, which is able to monitor and assist the trainee laparoscopy surgical movements. The developed prototype consists of a set of inexpensive sensors, namely an accelerometer, a gyroscope, a magnetome- ter and a flex sensor, attached to specific laparoscopic instruments[172]. Table 3.1 Review of serious games (continued) 40 Chapter 3. Serious games for health

Year Developer/Author Game Name Description 2011 Red Hill Studios and - A collection of therapeutic games for parkin- the School of Nurs- son disease patients to increase their balance ing at the University [177]. of California San Fran- cisco 2011 Scarle et al. Match-3 A serious game designed to combat childhood obesity. The Wii-mote is being used for a row- ing action which propels the vehicle forward, while direction is altered by leaning left and right on the Wii balance board [190]. 2011 Schnauer et al. - A chronic pain rehabilitation game provides multimodal interaction including full body motion capture by the use of Kinect, and other biosignal capture devices. Patients can manage their state and train physically on their own [191]. 2011 The Diablotines L’ Affaire BIRMAN This educational game aims to help patients treated by insulin to practice the technique of functional insulin therapy [208]. 2011 Urturi Z. - A serious game for Autism Spectrum Disor- der (ASD), oriented to first aid education: what to do in certain situations, basic knowledge about healthcare, medical specialities, etc [46]. 2011 Van Loon et al. - A set of computer games to help children with spastic cerebral palsy (CP) to loose the coupling between their hands. Patients were challenged to move both hands simultaneously in various phase relations [221]. 2011 Vazquez M. - A serious game to promote hand hygiene among health care professionals and citizens focusing specially on five moments for hand hygiene [222]. 2012 Applied Research As- HumanSim An immersive world where doctors and sociates, Inc. nurses train to learn the nuances of complex, unusual or other error-prone tasks until they become experts [10]. 2012 Chan et al. - A computer based surgical simulator which aims to train an ultrasound-guided needle placement which is a key step in a lot of radiological intervention procedures such as biopsy, local anesthesia and fluid drainage [32]. 2012 Cagatay et al. - A 3D game developed for speech and language disordered children. The game is used during the treatment process of Turkish children with language disorders [28]. Table 3.1 Review of serious games (continued) 3.4. Review of serious games for health 41

Year Developer/Author Game Name Description 2012 e-Learning Studios The iSpectrum This serious game aims to improve the work based social skills and relevant work skills of people with high functioning Autistic Spec- trum Disorders (ASD) and Asperger disorder [53]. 2012 EMCO3 MD Advisor A serious game that allows medical students to test their skills as future doctors such as performing the differential diagnosis in a virtual doctor’s office [58]. 2012 GENIOUS Interactive Voracy Fish A serious game for the upper limb rehabilitation. Movements are captured by Kinect, the player dives into a sea universe looking for some treasures and evolves while devouring other fish to become the strongest [81]. 2012 IKARE MUCOPlay A learning game to help caregivers, suffer- ers and families on cystic fibrosis. The game provides information about the right gestures and the means to validate their knowledge about cystic fibrosis and its care required [94]. 2012 Milo Foundation Miloland A serious game for children with a language delay, aged 9-11 years old but with a mental age of 5-6 years, i.e. starting literacy [143]. 2012 MIRROR project CLinIC-The Virtual The serious games focused on difficult com- Tutor and Think munication between nursing carer staff and better CARE-The patients/residents. These games aim to fos- Virtual Tutor ter reflection around difficult dialogues and to maximize learners ability to self-regulate their training [144]. 2012 Nike + Kinect Training Nike+ Kinect An exergame that combines fitness with gam- Training ing elements focusing on personalizing training and providing statistics [151]. 2012 SAIC, Inc. On-Line Inter- This serious game aims to train the medical active Virtual professionals with a set of training data with Environment different scenarios in a virtual hospital [187, (OLIVE) 192]. 2012 University Medical Air Medic Sky 1 The interactive bio-feedback game consists of Center Utrecht mini-games and lectures which describe the basic concepts required for efficient communication and teamwork resulting in patient safety [218]. 2012 Verduin et al. - A computer simulation designed for alcohol use disorders (AUDs) to practice relapse prevention skills [223]. Table 3.1 Review of serious games (continued) 42 Chapter 3. Serious games for health

3.5 Classiﬁcation of surveyed serious games for health

Figure 3.2: Classiﬁcations of serious games for health

Sawyer and Smith (2008) presented an interesting taxonomy on game for health categorized by a set of users (personal, professional, research/academic and public health) cross related with a set of serious games objectives (preventive, therapeutic, assessment, educational, and informatics) [189]. In our survey, we will deal with the game objectives but not with the users. We identify users as someone who play serious games in order to affect their personal health and therefore we do not mention the research and academia field of study as users. According to the divergence in scope and objective of our study, in this section we propose new parameters to categorize serious games for health in four different classifications. As it is illustrated in Figure 3.2, our classifications are based on three related subjects: serious game, health and player. Focusing on serious game subjects, we can classify by game purpose and game functionality, and for health subject, we can classify by state of disease. Finally, focusing on player subject, two types of players can be considered (player/non player and professional/non professional), and both are included into the same classification. More details of each classification are given below.

3.5.1 Classiﬁcation by game purpose

We consider here two categories of serious games for health according to its main purpose, Focused on Health (FH) and Focused on health acquisition and medical skills (FA). Since serious games are games not focused on entertainment [138, 189], we will not consider here games where wellness is obtained as a bonus in addition to entertainment, such as Dance Dance Revolution (DDR) [44]. Thus we consider serious games:

Focused on Health (FH), in this case, the main goal is health but the game is used • as a tool to pass on knowledge or skills. To use the capabilities of the game 3.5. Classiﬁcation of surveyed serious games for health 43

engine, various serious health contents are conveyed to players. For example, Fatworld [67], Re-Mission [206], Air medic sky1 [218] and many other games described in Section 3.4.

Focused on health acquisition and medical skills (FA), in this case, the game has • serious use for health purposes regarding a need of a virtual viewpoint or simulation to avoid or alleviate the risk, safety, budget, etc. Most of the games in this category are simulation games with virtual reality or augmented reality technology such as the Virtual dental implant training simulation program [25, 134], Emergency medical services for the disabled virtual environment (EMSAVE) [87, 226], Olive: 3D hospital training [187, 192], etc.

3.5.2 Classiﬁcation by functionality

Rego et al. [179] identified some criteria for the classification of serious games for health. We build upon their criteria (application area, interaction technology, game interface, number of players, game genre, adaptability, progress monitoring, performance feedback and game portability) and add some more characteristics (game engine, platform and connectivity) which have been consistently reported or published in descriptions of the games we surveyed (some of them have already been defined in Chapter 2, Section 2.2). The descriptions of our classification system are based on:

Application area or domain, which describes the part of the real world being mod- • eled by the software. In serious games for health, we will distinguish two main aspects: cognitive skills (Cog) such as memory, attention span, concentration and reasoning, and motor skills (Mot) such as general coordination or re-learning to walk after injuries.

Interaction technology has the different paradigms for establishing communica- • tion between humans and computers. Both hardware and software interfaces are included. Traditionally, mouse and keyboard have been used; newer means of interaction include virtual reality (using head-mounted displays), computer monitors, haptic or pseudo-haptic devices such as gloves or pens, or tracking devices. Webcams and web applications are also common. Patients can affect virtual objects in real-time using a variety of senses (vision, hearing and touching).

Game interface is related to the virtual world inside the game which can simulate • the real world in three-dimensions (3D) or provide a top-down scrolling and side scrolling for the perspective of a simpler world in two-dimensions (2D). Some techniques can be applied such as simulating the appearance of being 3D in isometric view with 2D objects which is called 2.5D or pseudo-3D. In some games a combination of hybrid 2D and 3D are employed.

Number of players concurrently using the world of the game. In general, we • distinguish single player games (for one person) and multiplayer games (for two 44 Chapter 3. Serious games for health

or more people).

Game genre is a categorization according to gameplay; we can distinguish ad- • venture, strategy, simulation, sports and puzzles, among others. In games for health, the games which evaluate coordination and movement are common in rehabilitation; other genres are also used for different tasks.

Adaptability (Yes/No) represents the adaptability of the difficulty according to • the skill of the player, in order to increase playability and enjoyment. Traditional games used to have a fixed difficulty level, which could either be programmed or chosen before the game started. In health, adaptability is an excellent trait be- cause it allows the patient to test and overcome his limits in a controlled manner.

Progress monitoring (Yes/No) is an advantage function for patient evaluation; • having logs of the patient actions inside the game can be an invaluable asset. We call this feature progress monitoring, since it allows the doctors to monitor the progress of the patients as a function of time.

Performance Feedback (Yes/No) are the indications of the game which deal with • showing the users their status and abilities. They allow patients to feel conﬁdent with their progress and to detect and ﬁx their failures. The feedback can be audio, visual or haptic.

Game portability (Yes/No) refers to being able to physically move the game hard- • ware. In particular, we distinguish between games located at a hospital or clinic from the ones that can be portable or used at home.

Game engine is a platform which provides commonly used game functionalities • and tools that allow the developers to create, edit design and functionalities themselves. The engine provides an API to access lower level functionalities and a set of predeﬁned models, materials and scenes [34]. Platform means the hardware the game runs on. This may include personal • computers (PCs), commercial game consoles (Nintendo Wii, Microsoft Xbox), portable consoles or custom hardware.

Connectivity is an ability to link to and communicate with other computer sys- • tems, electronic devices, software, or the Internet. Games might require an internet or network connection (online) or they may be played in standalone computers (ofﬂine).

3.5.3 Classiﬁcation by stage of disease

This classification is based on the stage of the disease the game is focused on. Following the classification proposed by Merrill [137], we can consider the four different categories, represented in Figure 3.3, together with the purpose of the game, as described below. 3.5. Classification of surveyed serious games for health 45

Figure 3.3: Serious games purpose related to stage of disease

Susceptibility stage: this is the ﬁrst stage, in which the person is still healthy. A se- • rious game in this stage helps the user to familiarize with monitoring procedures, and with illnesses that he/she could develop later in life.

Presymptomatic stage: in this stage, people still feel healthy although the illness is • already present. For example, the number of virus particles may still be too small to produce a response in the body, or a failing organ may still be able to cope with the added pressure of the illness with no external indications. The beginning of this stage may be discovered by the periodic check-ups mentioned above. In this stage, the speciﬁc illness is now known, and the chances of developing it are very high, so more focused serious games can be used to show the patients the relevant aspects of his illness and his treatment. EEG-based serious games [233] and PlayWithEyes [45] are an example.

Clinical disease stage: in this third stage, the symptoms of the illness are already • manifesting in the patients either as acute conditions or as the beginning of a chronic illness. During this phase, recognizable diseases manifest and diagnosis of the speciﬁc disease can be made. If the illness was not detected in the previous stages, serious games can be used to familiarize the patients with the expected progression of their illness, and the treatment procedures. Alternative treatments can also be shown using games. Games intended to be played by doctors or other medical staff usually focuses on this stage as well. Some examples are social skills [16], speech disorder children therapy [28] and improve bimanual coordination in children with spastic cerebral palsy [221].

Recovery, disability stage: in this last stage of the illness, two different outcomes • are possible: the illness may be cured, returning the patients to health or to another stage of susceptibility, or it may have serious effects on the patient’s health, making them unable to function at previous levels, that is, the illness may become chronic. In serious cases, a state of disability may occur. Serious 46 Chapter 3. Serious games for health

games in this stage normally deal with the rehabilitation procedure, or helping the patients cope with their disabilities such as neuropsychological rehabilitation [85], chronic pain rehabilitation [191], upper limb rehabilitation following stroke [27], after parkinson disease [177].

3.5.4 Classiﬁcation by player wellness (patients/non-patients)

Figure 3.4: Classiﬁcation of serious games for health by player

Regarding the deﬁnition of health, referred in Section 3.1, which covered both patients and non-patients, we classiﬁed serious games according to these two target player groups as follows (see Figure 3.4).

3.5.4.1 Serious games for patients

Serious games for patients can be classiﬁed according to their use into ﬁve different categories which are:

Health monitoring aims to keep an eye on patient health by monitoring the bio- • signals, such as heart failure telemanagement system [70], healthcare monitoring [68] and the u-health monitoring system [118]. Detection focuses on analysis or tracing of irregular symptoms of the patients. • For example unobtrusive health [133], eeg-based serious games [233] and Play- WithEyes [45]. Treatment or therapy is used to remedy a health problem. The examples are • match-3 [190], diagnosis and management of parkinson [13], social skills [16] and speech disorder children therapy [28]. Rehabilitation is a restoration of health and life skills after illness such as neu- • ropsychological rehabilitation [85], chronic pain rehabilitation [191] and upper limb rehabilitation following stroke [27].

Education for self/directed care increases understanding about the disease or health • problems and learning how to get and stay healthier with them. Some examples 3.5. Classiﬁcation of surveyed serious games for health 47

are re-mission [206], serious game for diabetes [149], ﬁrst aid education for autism spectrum disorder [46] and cognitive training for alzheimer [92].

3.5.4.2 Serious games for non-patients

Serious games for non-patients can be classiﬁed according to their use into three different categories which are:

Health & wellness games focus on lifestyle issues and their relationships with func- • tional health. We have surveyed the serious games regarding to the Alameda county study [89] which suggested that people can improve their health via i) exercise, ii) enough sleep, iii) maintaining a healthy body weight, iv) lim- iting alcohol use, and v) avoiding smoking. Some examples are sensory gate- ball game [106], dancing in the streets (DITS) [36], ﬁtness adventure [113], virku [219] and MoFun circus [156]. Training and Simulation for professional are serious games used as learning and • practicing tools for health professional. Some examples are HumanSim(Preview) [10], virtual dental implant training simulation program [25, 134], nursing and midwifery [199], pulse: the virtual clinical learning lab [5, 25] and emergency medical services for the disabled (EMSAVE) [226, 87]. Training and Simulation for non-professional are games used by laypersons to • learn to improve their healthcare. In this group there are games such as fatworld [67], the food detectives ﬁght BAC game [211], hand hygiene training [222] and nutri-trainer [78]. 48 Chapter 3. Serious games for health

3.6 Results

In this section, we present the surveyed games with respect to the proposed classifications. To better present the results, we create Table 3.2 that collects the information of games designed for patients and non-patients. All tables have the same column structure. From left to right, column (1) contains the name or the author of the game together with the publication year and column (2), presents the disease related to the game (or general health if no specific disease is mentioned). Column (3) presents the purpose, following the classification of Section 3.5.1, where FE represents focused on entertainment, FH focused on health and FA focused on health acquisition and medical skills. From columns (4) to (15), we present the information according to the classification by functionality (see Section 3.5.2). Column (4) is the application area which can be motor (Mot) or cognitive (Cot). Column (5) shows the tool that players use to connect to the game. Column (6) is game interface that can be 2D or 3D. Column (7) represents the number of players: single or multi player. Column (8) is the game genre that can be action, puzzle, simulation, etc. From columns (9) to (12), we represent adaptability, progress monitoring, feedback and portability, respectively. These columns are filled with three symbols (p, x and -), where p, where Y and N represent yes and no, and (-) represents that this feature is not mentioned in the game description. Columns (13) and (14) represent the engine/tool and the platform, respectively. Finally, column (15) shows connectivity that can be online (On) or offline (Off). In Ta- ble 3.2, rows have been grouped according to the classification by player wellness for patients (see Section 5.4.1), from top to bottom, health monitoring, detection, treatment, rehabilitation, and education for self/directed care (self-education). We continue with the classification by player wellness for non-patients (see Section 3.5.4.2). In this case, we consider three different categories, from top to bottom, health & wellness, training and simulation for professional and finally, training and simulation for non-professional.

3.6. Results 49 Connectivity

On On Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Platform PC PC PC PC PC PC PC PC PC PC PC PC Wii Mix

iPad Xbox

Mobile

/ tool Engine ------J2ME D-ﬂow Director Unity3D Unity3D Unity3D

Tailor-Made Tailor-Made Portability

x x p p p p p p p p p p p p p p p Feedback

- - - - - x p p p p p p p p p p p monitoring

------Progress p p p p p p p p p p p Adaptability

- - - p p p p p p p p p p p p p p Genre RPG RPG RPG Action Action Action Action Action Action Action Puzzle Puzzle Puzzle

Exergame Adventure Adventure Simulation Players Multi Single Single Single Single Single Single Single Single Single Single Single Single Single Single Single Single

atients) Interface

(Patients) 2D 2D 3D 2D 3D 3D 2D 3D 2D 3D 3D 3D 3D 3D 3D 3D

tool Interactive VR Mix EEG EEG Touch WiiFit Kinect Mouse Mouse Mouse Mouse Novint Camera Detection (P External Wiimote

Treatment Keyboard

VR Headset

area Application Cog Cog Cog Cog Cog Cog Cog Cog Cog Cog Cog Cog Mot Mot Mot Mot

Mot / Cog Purpose FA FA FH FH FH FH FH FH FH FH FH FH FH FH FH FH FH 3.2: Classiﬁcation and comparison of health games from our survey

Table Disease Alzheimer Neurofeedback Neurofeedback Eyes test Cancer Chronic pain Quit smoking Quit smoking Parkinson Neuro- disability Autism Cockroach phobia Autism Obesity Cerebral palsy Speech disorder Autismtrum spec- (2012) for Autisme and Duskin al. (2010) al. (2010) al. (2009) department of (2008) and Narasimhan al. (2012) and Gaggi (2011) Tomorrow (2006) al. (2011) al. (2011) et al. (2010) - Dancing et al. (2010) - Brain Chi

Loon et al. (2011) Author Robot et Microsoft (2011) (2004) Mckanna et Wang Wang De Bortoli Johnston, E Believe in Hatﬁeld D. Vermont Atkinson Bartolome et Finkelstein et Botella et Lakeside Center Scarle et Van Cagatay et e-Learning Studios health (2008) (2010)

50 Chapter 3. Serious games for health Connectivity

On On On On Off Off Off Off Off Off Off Off Off Off Off Platform PC PC PC PC PC PC PC PC PC PC PC PC iPad Multi

Android Xbox 360

/ tool Engine ------oolKit XNA XNA J2SE Flash ORGE Unity3D Unity3D

ART

Tailor-Made Portability

x p p p p p p p p p p p p p p p Feedback

- x p p p p p p p p p p p p p p monitoring

- Progress p p p p p p p p p p p p p p p Adaptability

x p p p p p p p p p p p p p p p Genre Mix RPG Action Action Action Action Action Action Action Action Puzzle Puzzle

Exergame Adventure Adventure Adventure Players

Multi Multi (Patients) Single Single Single Single Single Single Single Single Single Single Single Single Single Single Interface

2D 3D 3D 2D 2D 2D 2D 3D 3D 3D 3D 3D 3D 3D

2 / 3D 2 / 3D

tool Interactive VR Rehabilitation HMD

Touch Kinect Kinect Kinect Mouse Mouse Mouse Sensor Sensor Mobile

Webcam Webcam Webcam Wiimote

area Application

Cog Cog Cog Cog Cog Mot Mot Mot Mot Mot Mot Mot Mot Mot Mot Mot Purpose FH FH FH FH FH FH FH FH FH FH FH FH FH FH FH FH in- re-

injury Disease habilitation Knee surgery Stroke Stroke Stroke Stroke Stroke Wrist Brain health organizational problem Neuroillness General health Upper limb Parkinson Chronic pain Upper limb Language disabilities jury (2012) (2010) al. (2011) al. (2009) al. (2009), Catch al. (2009), Rabbit al. (2009), Virtual al. (2009), Whack a al. (2009), Arrow at- al. (2011) al. (2010) (2011) Classiﬁcation and comparison of health games from our survey (continued) oundation (2012)

et al. (2009) Hill Studios (2011) 2 Author task vibraphone chase mouse Edheads (2007) Burke et Burke et Burke et Burke et Burke et Deponti et Tost Fishing Cactus Grau et Lin JK. Moya et Red Schnauer et GENIOUS Interactive Milo F tack Table

3.6. Results 51 Connectivity

- -

On On On On Off Off Off Platform PC PC PC PC PC PC PC PC

Mobile

/ tool Engine ------

Flash Flash

Torque Portability

p p p p p p p p p Feedback

x p p p p p p p p

monitoring

Progress p p p p p p p p p Adaptability

x p p p p p p p p Genre Quiz Action Puzzle Puzzle

Adventure Adventure Adventure Adventure Simulation Players Single Single Single Single Single Single Single Single Single

(Patients) Interface

3D 2D 2D 3D 3D 2D 2D 3D 2D

tool Interactive

Mouse Mouse Mouse Mouse Mouse Mouse Mouse Mouse Mobile area

Self-Education Application

Cog Cog Cog Cog Cog Cog Cog Cog Mot Purpose

FH FH FH FH FH FH FH FH FH Disease Cancer General health Diabetes Alzheimer Diabetes Diabetes Autism Cystic ﬁbrosis AUDs (2011) al. (2011) al. (2011) Spil (2011) et al. (2012) (2012) Classiﬁcation and comparison of health games from our survey (continued)

2 Author Hopelab (2009) CCCP (2011) Fuchslocher et Imbeault et Nauta and The Diablotines Urturi Z.(2011) IKARE Verduin Table

52 Chapter 3. Serious games for health Connectivity

On On On On On On On On On Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Platform PC PC PC PC PC PC PC PC DS DS DS DS Wii Wii Wii Wii Wii Mix Mix Mix Multi

Mobile Mobile

Xbox360 Xbox360

/ tool Engine ------

Flash Flash Flash SMAC

Unity3D Portability

- x x x p p p p p p p p p p p p p p p p p p p p p Feedback

------x x x p p p p p p p p p p p p p p p p monitoring

- - - - - Progress x p p p p p p p p p p p p p p p p p p p Adaptability

- - x x x p p p p p p p p p p p p p p p p p p p p Genre Mix Sport Action Action Action Puzzle Puzzle Puzzle Puzzle Puzzle Puzzle

Exergame Exergame Exergame Exergame Exergame Exergame Exergame Exergame Exergame Exergame Exergame Exergame Exergame Adventure Players

Multi Multi Multi Multi Multi Single Single Single Single Single Single Single Single Single Single Single Single Single Single Single Single Single Single Single Single Interface

2D 3D 2D 2D 2D 2D 3D 2D 2D 2D 3D 3D 3D 2D 3D 2D 2D 3D 3D 2D 3D 2D 3D 3D 3D tool

wellness (Non-patients) Interactive Mix Mix RFID Cycle Kinect Kinect Kinect Kinect Mouse Mouse Mouse Mouse Sensor Sensor Button Button Button Camera Camera balance board balance board Balance board Balance board Balance Board Accelerometer

Wii Wii Wii Wii Wii

area Health & Application

Cog Cog Cog Cog Cog Cog Cog Mot Mot Mot Mot Mot Mot Mot Mot Mot Mot Mot Mot Mot Mot Mot Mot Mot Mot Purpose FH FH FH FH FH FH FH FH FH FH FH FH FH FH FH FH FH FH FH FH FH FH FH FH FH Ears

exercise Disease General health General health Brain health Brain health Chi Face General health General health Brain health Brain health General health General health General health General health General health Eyes and health over-binge General health General health General health General health Obesity General health General health General health Inc. Centre Centre raining (2009) Co., Ltd. (2006) (2009) Inc. (2010) Studios, Ltd. Entertainment LLC (2010) (2007) Games (2009) (2004) Kinect T al. (2010) .l. (2009) Leikas (2009) (2009) + al. (2009) Classiﬁcation and Comparison of Health Games from our survey (continued) Entertainment GmbH

2 Author (2010) (2007) (2012) (2009) (2009) (2007) Montreal Science Respondesign Nintendo (2005) Gameloft (2006) McGIll University Intelligent System Nintendo EAD Nordic Innovation SEGA (2007) BBG Blitz Games Collision Studios Kim et Laikari A. Lightning Fish Raylight S.r Succubus Interactive Vtnen and Anchor Bay Clawson et Electronic Arts, HopeLab (2010) Ubisoft Divertissements, Zumba Fitness Nike (2010) (2004) Table

3.6. Results 53 Connectivity

On On On On On On On On On On Off Off Off Off Off Off Off Off Off Off Off Off Off Off Off Platform

PC PC PC PC PC PC PC PC PC PC PC PC PC PC PC PC PC PC PC PC PC PC PC PC

iPad Multi

/ tool Engine ------orce 3D XNA Flash Flash Unreal Unreal Torque TruSim TruSim NeoAxis Unity3D

SecondLife SecondLife Tailor-Made Tailor-Made Tailor-Made Tailor-Made

Star F Portability

- - p p p p p p p p p p p p p p p p p p p p p p p p Feedback

- - - p p p p p p p p p p p p p p p p p p p p p p p monitoring

- Progress p p p p p p p p p p p p p p p p p p p p p p p p p Adaptability

- - - - - p p p p p p p p p p p p p p p p p p p p p Genre RPG

Adventure Simulation Simulation Simulation Simulation Simulation Simulation Simulation Simulation Simulation Simulation Simulation Simulation Simulation Simulation Simulation Simulation Simulation Simulation Simulation Simulation Simulation Simulation Simulation Simulation Players

Multi Multi Multi Multi Single Single Single Single Single Single Single Single Single Single Single Single Single Single Single Single Single Single Single Single Single Single Interface

3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 2D 3D 3D 3D 3D 3D 3D 3D

tool ec Interactive Touch Mouse Mouse Mouse Mouse Mouse Mouse Mouse Mouse Mouse Mouse Mouse Mouse Mouse Mouse Mouse Mouse Mouse Mouse Mouse Mouse Mouse Mouse VoiceR BioFeedB

Laparoscopic for Professional (Non-patients)

area Application Cog Cog Cog Cog Cog Cog Cog Cog Cog Cog Cog Cog Cog Cog Cog Cog Cog Cog Cog Cog Cog Cog Cog Cog Cog Cog

Training Purpose FA FA FA FA FA FA FA FA FA FA FA FA FA FA FA FA FA FA FA FA FA FA FA FH FH FH Health

illness Disease eplacement Triage Dental Injuries Diabetes Pregnancy General health General health General health General health General health General health General health General health General health General health General health General health General health General health General health General health General health General health Respiratory Knee R Maternal Child Center Univer- (2010) (2008) (2012) Learning, Inc. (2010) (2008) Murphy (2008) al. (2011) , Ltd. (2007) , Ltd. (2010) (2012) esearch Associates, (2008) - Triage al. (2011) al. (2012) al. (2010) (2010) - Patients Res- Heroes, Inc. (2009) Imagination Software al. (2008) et al.(2010) Classiﬁcation and Comparison of Health Games from our survey (continued) University (2009)

2 Author Trainer (2010) Inc. (2012) (2010) BreakAway Anderson C. Glasgow Caledonian sity (2008) Imperial College Mili et Sliney and TruSim Keele Virtual BreakAway Innovation in KTM Advance Miller (2010) Sabri et Skills2Learn, Ltd. TruSim cue Vidani Visual Diehl et Queiros et Applied R Chan et EMCO3 (2012) MIRROR project SAIC, Inc. University Medical Utrecht (2012) Table

54 Chapter 3. Serious games for health Connectivity On On On On On On On On On On On On On Off Off Off

PC Platform PC PC PC PC PC PC PC PC PC PC PC PC PC Multi

Mobile

Pocket

/ tool Engine - - - -

WPF Flash Flash Flash Flash Flash Flash Flash Flash Flash Flash Flash Portability

p p p p p p p p p p p p p p p p Feedback

- - - x x x x x x p p p p p p p monitoring

- - - Progress x x p p p p p p p p p p p Adaptability

- - - x x x p p p p p p p p p p Genre RPG Quiz Action Action Action Action Action Puzzle Puzzle Puzzle

Strategy

Adventure Adventure Adventure Adventure Simulation Players

Single Single Single Single Single Single Single Single Single Single Single Single Single Single Single Single Interface 2D 3D 2D 2D 2D 2D 2D 3D 2D 2D 2D 2D 3D 2D

2D / 3D 2D / 3D

tool Interactive

Touch Touch Mouse Mouse Mouse Mouse Mouse Mouse Mouse Mouse Mouse Mouse Mouse Mouse Mouse Button

area Application for Non-professional (Non-patients)

Cog Cog Cog Cog Cog Cog Cog Cog Cog Cog Cog Cog Cog Cog Cog Cog Purpose FH FH FH FH FH FH FH FH FH FH FH FH FH FH FH FH

Training Disease inﬂuenza Flu Flu AIDS H1N1 Obesity Diabetes Nutrition Epidemics Cardiac arrest General health General health General health General health General health General health Avian (2011) BFM / Lab (2009) (2006) Agency of Canada Games (2009) ation Centre (2007), ation Centre (2007), (2007) Games LLC (2009) M. (2011) Bros. Entertainment, Inc. (2009) al. (2010) eacher Education Program Classiﬁcation and Comparison of Health Games from our survey (continued)

Safety Education (2008)

2 Author (2004) (2011) (2008) MIT T Archimage, Inc. Janomedia (2006) Nordic Innov Nordic Innov Fatworld.org Food Warner Learning Games Persuasive QOVEO RANJ Serious Gago et Association RMC Public Health Vazquez MC Urho Valion Energiasummaaja Table 3.7. Discussion 55

3.7 Discussion

The information collected from Table 3.2 has been used to compare the characteristics of the surveyed serious games with respect to different parameters. Below, from Fig- ures 3.5 to 3.21, we present the obtained results by a graphical summary together with a brief description.

Figure 3.5: (a) Distribution of surveyed serious games according to type of players and (b) number of surveyed serious games for patients and non-patients according to the year of publication

Figure 3.5 shows the production of games according to the player (patients, non- patients). In Figure 3.5(a) we illustrate the number of games for patients with respect to games for non-patients. We observe that serious games for non-patients are about twenty seven percent more common than serious games for patients. In Figure 3.5(b), we present the evolution of game production, according to the year of publication, distinguishing between games for patients and non-patients. We will see that the maximum of the production is reached in 2010. Then in 2011, the interest on serious games for patients increases. The intersection point in 2010-2011 indicates that the interest in serious games for patients may be higher than that of serious games for non-patients in the future. Focusing on the objectives of the developed serious games (Figure 3.6), we can see that their objectives are quite varied, but an emphasis can be seen on professional training and health & wellness (almost a quarter) followed by serious games for non professional training, rehabilitation, treatment, education and detection respectively. We classiﬁed serious games for learning purposes (professional training, non professional training and self education) from medical care as illustrated in Figure 3.7. We see that number of serious games for learning in our survey is almost half of them, see 56 Chapter 3. Serious games for health

Figure 3.6: (a) Distribution of surveyed serious games according to its objective

Figure 3.7: (a) Distribution of surveyed serious games focused on learning according to its objective, medical care or learning and (b) number of surveyed serious games focused on learning objective according to the year of publication

Figure 3.7(a). The trend of serious games focused on learning objective is moving up even though the graph shows the ﬂuctuation of the number of serious games published per year until now, see Figure 3.7(b).

Figure 3.8: Distribution of surveyed serious games with regard to disease

Figure 3.8 presents the distribution of serious games according to the disease for which they have been designed. The chart shows the wide variety of the use of seri- 3.7. Discussion 57 ous games in many different diseases. About forty percent of the serious games are designed for general health. Within speciﬁc diseases, stroke is the most commonly addressed disease followed by diabetes, brain health and autism respectively. We notice that most of serious games for health & wellness and training for non-professional are made for general health purposes. On the other hand, there are no serious games for general health presented in patients detection and treatment purposes.

Figure 3.9: (a) Distribution of surveyed serious games with regard to brain related disease and (b) number of surveyed serious games focused on brain disease according to the year of publication

Figure 3.9 presents the production of the serious games focusing on the diseases related to the brain which amount to a quarter of all games, see Figure 3.9(a). We observe that the interest in brain related diseases became higher during the year 2009 and the number of games published fluctuates every year. Focusing on the game purpose by the classification in Section3.5.1, Figure 3.10 presents the distribution of serious games according to its purpose. In the top chart, Figure 3.10(a) we can see that three quarter of serious games focus on health while a quarter focus on health acquisition and medical skills; 88 percentage (23 out of 26) of serious games for professional training are in this group. Furthermore, we classified game purpose according to the player (patients and non-patients) as shown in Fig- ures 3.10(b) and (c); we will see that almost all serious games for patients are focused on health. On the other hand, the number of serious games for non-patients focused on health are approximately thirty percent more than the one focused on health. Focusing on application area, Figure 3.11 shows that both motor and cognitive abilities are well represented in our survey. An amount of two third of serious games are for cognitive improvement while one third serious games for motor improvement and only one of the hundred and nine surveyed has been designed to improve both cognitive and motor skills. Almost all (50 out of 51) of the serious games for education purposes (self education, training for both professional and non-professional) are focused on cognitive skills. On the other hand two thirds (18 out of 25) of the 58 Chapter 3. Serious games for health

Figure 3.10: (a) Distribution of surveyed serious games according to its purpose, (b) distribution of surveyed serious games for patients according to its purpose and (c) distribution of surveyed serious games for non-patients according to its purpose

Figure 3.11: (a) Distribution of surveyed serious games with regard to application area, (b) distribution of surveyed serious games for patients with regard to application area and (c) distribution of surveyed serious games for non-patients with regard to application area

serious games for health & wellness purposes are designed for motor skills. From Fig- ures 3.11(b) and (c) which present the number of application areas for patients and non-patients, we will see that the number of motor improvement games for patients (16) and non-patients (17) is similar. Figure 3.12 presents the distribution of surveyed serious games according to interaction tools. The results are quite varied, however the standard mouse interface is used in about half of the games, followed by Wii peripheral, Kinect, camera and touch screen respectively. Additionally, if we focus on Figure 3.13 where we show the distribution of games 3.7. Discussion 59

Figure 3.12: Distribution of surveyed serious games according to interaction tool

according to the top four interaction tools and the year of publication we will see that mouse is in the leader position except in the year 2011 which was led by other tools.

Figure 3.13: Distribution of surveyed serious games according to top four interaction tools and the year of publication

Focusing on the graphic user interface, Figure 3.14(a) shows that 2D and 3D interfaces are both well represented in the surveyed games. However, the number of 3D games is twenty ﬁve percent higher than 2D games. We also observe that almost all (24 out of 26) of the interface for professional training are 3D. The 3D interface also shows a major presence in both serious games for patients and non-patients which can be seen in Figures 3.14(b) and (c). Figure 3.15 presents the evolution of both 2D and 3D interface in our surveyed serious games according to the year of publication. We see that the trend of 3D interfaces is growing up in average since 2007, with the maximum value on year 2010. At the same time, the 2D interface graph ﬂuctuated from 2007 to 2009 then decreased since year 2009. We can say that the interest in 3D interfaces overcame 2D interfaces since 2010. Figure 3.16 reports the distribution of our surveyed serious games according to the number of players. We will see from Figure 3.16(a) that most of the games (9 out 60 Chapter 3. Serious games for health

Figure 3.14: (a) Distribution of surveyed serious games with regard to interface, (b) distribution of surveyed serious games for patients with regard to interface and (c) distribution of surveyed serious games for non-patients with regard to interface

Figure 3.15: Number of 2D and 3D interfaces of our surveyed serious games according to the year of publication

of 10) are currently single player. Multi player is presented mostly in the health & wellness games, followed by serious games for professional training. Figures 3.16(b) and (c) show that multi player games are represented in serious games for non-patients rather than serious games for patients. Figure 3.17 illustrates the distribution of the surveyed serious games according to game genre. Figure 3.17(a) shows that the simulation and action genres dominate, followed by puzzle, exergame and adventure respectively, although there is a large variety of other genres. Almost all of the simulation genre (25 out of 27) in our survey are created for non-patients. As well as the action genre which most of them (16 out of 23) are created for patients. Almost all of serious games for professional training are 3D 3.7. Discussion 61

Figure 3.16: (a) Distribution of surveyed serious games with regard to its number of players, (b) distribution of surveyed serious games for patients with regard to number of players and (c) distribution of surveyed serious games for non-patients with regard to number of players

Figure 3.17: (a) Distribution of surveyed serious games with regard to its genre, (b) distribution of surveyed serious games for patients with regard to game genre and (c) distribution of surveyed serious games for non-patients with regard to game genre

simulation genre (24 out of 26). Although action games are used for many objectives (treatment, health & wellness, etc.) the most important one is rehabilitation, as well as an exergame which is the prominent genre on serious games for health & wellness. From Figure 3.17(b), the top three game genres for patients are action, adventure and puzzle. From Figure 3.17(c) we can see that the top three game genres for non-patients are simulation, exergame and puzzle. 62 Chapter 3. Serious games for health

Figure 3.18: Distribution of surveyed serious games according to (a) adaptability, (b) progress monitoring, (c) performance feedback and (d) portability

Figure 3.18 presents the statistics on our survey (from left to right) in adaptability, progress monitoring, performance feedback and portability respectively. All are important features, with useful characteristics, and are included in the majority of the serious games as mentioned in Section 3.5.2. These charts show that each of these functionalities is considered as major (more than three quarters of the surveyed serious games implement them).

Figure 3.19: Distribution of surveyed serious games according to game engine

About the production tools, Figure 3.19 shows a wide variety of engines used in surveyed serious games, although this is not an often reported characteristic. Flash is the most often used engine, followed by tailor made engines and Unity3D. Figure 3.20 presents the distribution of our surveyed serious games according to game platform. From Figure 3.20(a) we will see that there are few used platforms, most of the games (two thirds) have been designed to run on personal computers followed by console and handheld platforms. There are some serious games which were developed for use in multiple platforms, which appear in games for non-patients rather than in games for patients. From Figures 3.20(b) and (c), personal computers are the most often used platform in both serious games for patients and non-patients. The console and multiple platform are more common in serious games for non-patients while handheld platform is present more in serious games for patients. 3.7. Discussion 63

Figure 3.20: (a) Distribution of surveyed serious games with regard to game platform, (b) distribution of surveyed serious games for patients with regard to game platform and (c) distribution of surveyed serious games for non-patients with regard to game platform

Figure 3.21: (a) Distribution of surveyed serious games with regard to connectivity, (b) distribution of surveyed serious games for patients with regard to connectivity and (c) distribution of surveyed serious games for non-patients with regard to connectivity

Finally, Figure 3.21 illustrates the distribution of our surveyed serious games according to internet connection. From Figure 3.21(a) we will see that the internet connection is used in one third of the games surveyed. Figures 3.21(b) and (c) show that most of the internet connectivity present in serious games for non-patients (32 out of 41). Most of serious games for non-professionals (13 out of 16) are an online games which aim to be seen by laypersons. On the other hand, most (11 out of 13) of serious games for treatment are offline. Most of serious games for rehabilitation, patients self education and health & wellness purposes also are offline due to the size of the game which might be huge and difficult to play smoothly in an online environment or the games may need some special external tools to perform the tasks given. We did a statistical analysis of the different variables using statistical analysis soft- 64 Chapter 3. Serious games for health ware (SPSS) to see if certain characteristics correlated with certain game requirements. In particular, we considered that the type of user (patients/non-patients), stage of disease, disease, purpose, application area, adaptability, progress monitoring and feedback were chosen a-priori by medical staff, while the interactive tool, the interface, connectivity, players, genre, portability, engine and platform were chosen by the game designers. However, there are no statistically significant correlations between different variables. Therefore, we consider that in principle, games of different characteristics can be applied to the different requirements.

3.8 Conclusion

We have studied one hundred-and-nine serious games from academic and commercial environments (including a variety of online games) dealing with health in a broad sense, including medicine, nursing, health care and physical exercise. The games have been classified according to their main purpose (entertainment, teaching or health), stages of the disease being treated (health monitoring, detection, treatment, rehabilitation and education) and the type of users of the system (general population, patient, health professional). Additionally, fifteen criteria dealing with the game technology have been selected for a fine-grain classification: application area, interaction technology, platform, game engine, interface, portability, connectivity, adaptability and genre, number of players, performance feedback, progress monitoring and health objective. Although for most of these criteria there were a wide range of possibilities, which have been explored by at least one game, our results indicate that one of the possibilities in each category has been predominantly chosen by game designers. The average game can therefore be summarized as a portable PC game, using mouse interaction and including progress monitoring, performance feedback and adaptability. The most common genres were simulation and action (which account for half of the games). However, we found little correlation among these characteristics. As an example, only seven of the one hundred-and-nine games reviewed have all the characteristics of the average game described. The variability of games is quite large in many aspects. The presented description and classification of games can be useful for researchers developing new games by raising the awareness of the different possibilities. While we cannot make hard predictions on the future of serious games for health, we expect two different directions: new, innovative games exploring the rest of the parameter space of our classification, and more classic games concentrating on the most commonly used features. As a future trend, we expect that the 3D interface in both PCs and handheld platform with online connectivity will dominate the serious games for health market, given the increasing capability of handheld devices with PCs-like functionality. We also expect the real-time interaction between therapist and player will lead to a powerful tool for patients recovery and treatment. CHAPTER 4 LISSA, a Life Support Simulation Application

4.1 Introduction

A key point to carry out the work of this thesis is the selection of an application area to focus our study on it. We selected health applications focused on cardiopulmonary resuscitation (CPR). CPR is a first aid key survival technique used to stimulate breathing and keep blood flowing to the heart. This technique is used to recover from cardiac arrest. The American Heart Association (AHA) states that nearly 383,000 out-of-hospital sudden cardiac arrests annually occur, and 88 percent of them occur at home. Effec- tive CPR administration can significantly increase the chances of survival for victims of cardiac arrest that take place outside of hospital. Since 1960, when Kouwenhoven published an article stating that anyone, anywhere, could perform CPR [108], providing CPR has become an essential competency not only for expert or professional but also for laypersons. The CPR protocol consists in a set of procedures that have to be applied in a correct order and in an specific way. These procedures determine, among others, how to assess an unconscious person by checking the airway, breathing and circulation (known as ABCs, [73]) and how to do chest compression. To learn CPR different strategies have been proposed, such as traditional classroom, video self-instruction and computer-based programs [6, 24, 37, 72, 148, 178]. In this chapter we propose LISSA (LIfe Support Simulation Application), a serious game designed to teach and learn CPR. LISSA exploits video game technology to link in a single framework computer-based case simulations with e-learning functionalities.

4.2 Related work

In this section, we describe the basic CPR protocol and also previous work on applications and serious games designed to learn CPR.

4.2.1 The basic CPR ﬂowchart

The CPR procedures that have to be applied in an emergency case depend on different parameters such as the age of the patient, his initial situation, or the presence of medical devices. To know how to proceed in each case we can follow some of the 66 Chapter 4. LISSA, a Life Support Simulation Application graphic flowcharts provided by different organizations such as the European Resusci- tation Council (ERC) [63], or the Red Cross or the American Heart Association [6]. In these sites, we can found from the basic CPR flowchart to the more advanced one (see Figures 4.1 and 4.2). To present the basic CPR flowchart we are going to consider an emergency case where an adult patient is lying on the floor. To perform CPR we have to follow the flowchart represented in Figure 4.1. To start, the helper checks if the patient is con- scious by checking gently or shouting loudly. If the patient responds the helper can consider that the patient is fine and then only has to observe for new symptoms. If no symptoms appear the situation is controlled and no help is necessary. If the patient does not respond, the helper has to open the airway in order to give an easy posture to get air circulation. In this state, breath is observed. If the patient is breathing normally, the helper should arrange the patient posture to the recovery one to avoid suffocation and observe until patient is fine. If the patient is not breathing, the emergency call is urgently required and a sequence of 30 chest compressions and 2 breath deliveries maneuver has to be applied. This step is represented as CPR(30:2) sequence. This protocol changes if there is an automated external defibrillator (AED) nearby. In this case, we have to follow the audio and video commands provided by the AED until help arrives. Being CPR certified means having the ability to apply the CPR procedures (i.e. compression rate, compression depth, ventilation duration and ventilation volume) in the correct way. CPR is commonly taught in classrooms where an expert introduces the main CPR procedures and then practice is done with the support of manikins. This strategy requires the supervision of an expert who controls that the procedures are correctly applied. To practice CPR out of the classroom, computer-based media including serious games for CPR can be used.

4.2.2 Serious games and CPR

Different applications and serious games to learn CPR have been proposed in last years. Among them, there are video training applications such as Save-A-Life Simulator [136], an interactive online video simulation that tests the player knowledge of helping someone suffering from sudden cardiac arrest; and CPR & Choking [202], an application that provides instant information on how to perform CPR and how to aid a choking victim. These applications show a one minute video to present the latest recommenda- tions from the major international resuscitation organization including the American Heart Association and the International Liaison Committee on Resuscitation. There are also handheld applications such as, CPR Game [57], a cardiac arrest simulator on iOS platform focused on advanced CPR training guided by 2010 Advanced Cardiovascular Life Support guidelines; iResus [127], an application for smart phone, designed to improve the performance of an advanced life support provider in a simulated emergency situation; iCPR [40, 196], an iPhone application designed for both lay persons and healthcare professionals able to detect the rate of chest compressions performance by using the built-in accelerometer; M-AID [2], a ﬁrst aid application for mobile phones 4.2. Related work 67

European Resuscitation Council

www.erc.edu [email protected] Basic Life Support ⇢ Check response

- Shake gently - Shout loudly

⇢ If NOT responsive If responsive Open airway - Observe Check breathing - Get help if necessary

- Tilt head back and lift chin - Look - Listen - Feel - Take no more than 10 s.

⇢ If not breathing normally If breathing normally

Call 112 - Place in recovery position Deliver 30 chest - Get help compressions - Recheck breathing

Place your hands in the centre of the chest

Deliver 2 rescue breaths

- Seal your lips around the mouth - Blow steadily until chest rises - Give next breath when the chest falls

⇢ Continue CPR 30:2 until qualified help arrives

Published March 2007 by European Resuscitation Council Secretariat VZW, Drie Eikenstraat 661, 2650 Antwerp, Belgium Product reference: POSTER-2007-BLS_A0-EN Copyright European Resuscitation Council

Figure 4.1: CPR guideline provided by the European Resuscitation Council (ERC2010) 68 Chapter 4. LISSA, a Life Support Simulation Application

European Resuscitation Council

www.erc.edu [email protected] Automated External Defibrillation Check response - open airway - check breathing - call 112 Activate the AED Follow the voice prompts without delay.

Attach pads

Attach one pad below the left armpit. Attach the other pad below the right collar bone, next to the breastbone.

Stand clear

Ensure that nobody touches the victim whilst the AED is analysing the heart rhythm.

Deliver shock

Ensure that everybody is clear of the victim.

Start CPR immediately

Place your hands in the centre of the chest. Deliver 30 chest compressions. Seal your lips around the mouth. Blow steadily until the chest rises. Give next breath when the chest falls. Continue CPR. 30:2 Follow the voice prompts without delay. If the victim starts to breathe normally, stop CPR. If still unconscious, turn him into the recovery position.

Published March 2007 by European Resuscitation Council Secretariat VZW, Drie Eikenstraat 661, 2650 Antwerp, Belgium Product reference: POSTER-2007-AED_A0-EN Copyright European Resuscitation Council

Figure 4.2: CPR with automated external deﬁbrillation (AED) guideline provided by the European Resuscitation Council (ERC2010) 4.3. The LISSA framework 69

Name Platform Interaction tool Interface Connectivity Purpose JUST (2003) PC Mouse 3D Online Learning CPR simulator (2006) Handheld Mouse 2D Offline Testing M-AID (2007) Handheld Mobile Button 2D - Testing iCPR (2009) Handheld Accelerometer 2D Online Testing iResus (2010) Handheld Touch Screen 2D Online Testing AED Challenge (2011) PC Mouse 2D Online Learning CPR & Choking (2011) Handheld Touch Screen Video - Learning Staying alive (2011) PC Mouse 3D Online Testing CPR Game (2012) Handheld Touch Screen 2D Online Testing MicroSim-Prehospital PC Mouse 2D, Video Offline Learning, Testing (2012) Mini-VREM (2012) PC Kinect 2D Offline Testing Save-A-Life Simulator PC Mouse Video Online Learning (2012) LISSA (2013) Multi Mouse, Kinect 3D Online Learning, Testing

Table 4.1: Comparison of CPR applications (-) represents that this feature is not mentioned in the reference that uses yes or no questions to judge an ongoing situation giving to the user detailed instructions of how to proceed; and CPR simulator [119, 135], a set of CPR exercises including adult, child and infant CPR simulator that runs through the CPR sequence. In addition, some applications for PC platforms are Mini-Virtual Reality Enhanced Mannequin (Mini-VREM) [93, 195] which is a CPR feedback device with motion detection technology including Kinect, sensor and software speciﬁcally designed to analyse chest compression performance and provide real-time feedback in a simulation training setting, and AED Challenge [96], an application that provides online automated external deﬁbrillation and CPR skill practice and testing with realistic scenarios. Finally, in the serious games context, some games for CPR training are JUST [167], an immersive VR situation training system for non-professional health emergency op- erators, MicroSim Prehospital [110] designed for pre-hospital training on emergency medical services, and Staying alive (2011) [43], an online 3D simulator which provides a learning experience of saving a virtual patient from cardiac arrest in four minutes. Table 4.1 presents the main features of all reported methods. From left to right, we present the year of publication, the platform for which has been designed, the interaction tool that supports, the type of interface, distinguishing between 2D, 3D and video, the required connectivity and also the main purpose of the application. In the last row we also include the features of our proposed framework LISSA, which is presented in next section.

4.3 The LISSA framework

LISSA has been designed to introduce and increase the knowledge of CPR to any kind of public. Our framework (see Table 4.1) is 3D multi-platform and supports interaction via mouse and Kinect. The platform works online and its main objective is learning and 70 Chapter 4. LISSA, a Life Support Simulation Application testing. LISSA has been conceived considering both teachers and learners and with the idea to make teachers tasks easier. In this way, LISSA allows the creation of different scenarios with different characters, patient symptoms and environments. Moreover, the system provides automatic feedback to the learner which enhances the learning process. In this section, we present the main components of LISSA, how we deﬁne the CPR scenarios and also proposed score strategy.

4.3.1 The main components

The main components and functionalities of the LISSA framework are presented in Figure 4.3 and described below.

Figure 4.3: Main modules of LISSA framework

LISSA has been designed as an e-learning environment with all the actions turning around a CPR scenario that reproduces with 3D realism an emergency situation which requires CPR procedures. LISSA supports two type of users, instructors and learners. The CPR scenario is deﬁned by the instructor and presented to the learner as a test or problem. The learner solves the problem applying the CPR procedures in a game mode. LISSA evaluates the actions and assigns a ﬁnal score. All the learner’s actions are registered in a central data base allowing instructors to consult them in order to track the learning process. Instructors can use this information to recommend new scenarios and problems. To present the functionalities of the game we distinguish between instructors and learners. When the user is an instructor LISSA provides functionalities to:

Prepare CPR material by accessing the content creation module. The system differ- • entiates between exposition material and problems. Exposition material shows how to apply CPR, using a link to a document or a video. Problems are designed to practice CPR, they are presented as an emergency case. The instructor can modify some of the problem parameters such as the scenario, the victim or the helper. 4.3. The LISSA framework 71

Assign problems to the learners. The instructor can determine the problems and • the material he/she wants to assign to the learner. Each learner has assigned a workbook which contains a link to these problems.

Track learner work. Since the learner workbook also maintains the entered solu- • tions, the instructor can consult the learner progress.

When the user is a learner, LISSA provides functionalities to:

See exposition material. The learner can visualize the content selected by the • instructor as many times as he/she wants. Solve problems. The system reproduces a CPR emergency situation and the • learner applies the CPR procedures which are considered as the problem solution. The system evaluates the solution and assigns a ﬁnal score (see subsec- tion 4.3.3).

See feedback. For each solution the system creates a report with all the infor- • mation related to the proposed solution, the learner can consult it to follow his progress.

Two of the main modules of our framework are the content creation, which reproduces the CPR scenario, and the correction module, which determines the score of the learner. Both are described in detail in next sections.

4.3.2 The content creation module

The content creation module creates the 3D scene with all the elements (actors, medical devices, etc.) that have to be applied to reproduce the situation where the CPR is required. The learner interacts with this scene in order to recover the victim. To create the scene we have to determine how to model the CPR scenario, how to graphically reproduce the scene and how to deﬁne the interaction between the learner and the scene.

4.3.2.1 Modeling the CPR scenario

The CPR scenario has been modelled as a state machine which consists of a set of states and a collection of transitions, which represent some kind of actions for each state. Its purpose is to describe how an object can change its state over time in response to the environment and events that occur. The use of state machines in videogames is promoted by many developers due to their robust nature as they are easy to test and modify [90]. 72 Chapter 4. LISSA, a Life Support Simulation Application

Figure 4.4: (a) CPR ﬂowchart for an adult patient [63] and (b) proposed UML state machine to model the CPR ﬂowchart 4.3. The LISSA framework 73 UML state machine of basic cpr procedure (without AED machine) Figure 4.5: 74 Chapter 4. LISSA, a Life Support Simulation Application

Using a state machine requires a prior planning of all game states and actions. In our case, this information is known and very well-defined by the CPR guideline provided by the European Resuscitation Council [63] (see Figure 4.1). To illustrate the relationship between the CPR flowchart and the proposed state machine modelled with the Unified Modelling Language (UML), we present both of them in Figure 4.4(a) and (b), respectively. For the sake of simplicity we show a high-level design of the UML diagram. To create the applicable state machine, more details need to be completed. The full version of state machine for Basic CPR procedure (without AED equipment) is shown in Figure 4.5. A main advantage of the proposed design is that different scenarios can be easily created by simply modifying parameters such as where the emergency situation has occurred, who is the victim, position of the patient and success percentage. These parameters will be used to determine the level of difficulty of the problems as we will see in Section 4.3.3.

4.3.2.2 CPR scene reproduction

Once the state machine has been deﬁned, we have to create the scenes and the animations corresponding to the states and the actions (or transitions) respectively. We have to create a scenario-based structure to reproduce a CPR emergency situation where learners will apply their CPR knowledge and skills to recover the victim from the presented situation. Our aim is to reproduce realistic scenes, and for this reason 3D characters and 3D scenes have been chosen.

Figure 4.6: Some of the models stored in the LISSA database and used to create CPR scenes: (a) football ﬁeld, (b) ambulance, (c) hospital, (d) helper and victim models and (e) some of the supported devices.

In Figure 4.6, we show some of the situations that have been considered. In the first row, we illustrate three different scenarios, a football field, an ambulance and a hospital. In the second row, we illustrate some of the actors, the patient and the 4.3. The LISSA framework 75 helper, and also some of the devices that can be used. To create these scenes, several art application have been used. The modelling and animation procedures have been finished with Autodesk 3ds Max. Most of the textures are solid color and real photo. In order to create some advanced textures, such as crease, some real image have been used in accordance with some modelling techniques, e.g. sculpturing, within Z-brush application. To ensure real-time interaction and preserve image realism we create scenes where the number of polygons for one real-time render scene, including characters, equipments and environment, is between 40,000 - 200,000 polygons [182]. In addition, most of the texture images have a jpg format which can be lossy compressed. On the other hand, in order to represent the transparent objects we use tif format since it supports alpha channel. The animations have been created with Autodesk 3ds Max and then have been assigned to the animation component function on Unity3D which allows the playback animations be controlled by script. The set of animations corresponding to the button on game interface will be played when enabled. Finally, to illuminate the scenes and the objects we use the lighting functionalities of Unity3D game engine [216]. More details are described in Chapter 5. To support the creation of different CPR scenarios we maintain in a central database models of the possible 3D objects and their animations related to CPR actions (see Fig- ure 4.6). To create a new problem, the instructor has to select from the database the elements of the scene. We also allow the instructor to modify some of the parameters related to the objects, such as the position of the victim (lateral or supine), the gender of the patient (male or female) and the location. These parameters can also be randomly determined. In Figure 4.7, we show a screen-shot of the interface designed to enter these parameters and Figure 4.8, shows some of the scenarios, from left to right, a reception area of a hospital, a football field with AED machine and in an ambulance. The capability of our system to create scenarios with different levels of difficulty provides advantages to both instructors and players. On the one hand, instructors can prepare exercises of different levels of difficulty according to the learner profile. On the other hand, experienced players can deal with a variety of situations. 76 Chapter 4. LISSA, a Life Support Simulation Application

Figure 4.7: Interface designed to determine the parameters that deﬁne the CPR scene

Figure 4.8: CPR scenarios automatically created with the objects from the content creation database of LISSA

4.3.2.3 User Interaction

Different strategies can be applied to interact with our CPR virtual world [3, 1]. During the graphic user interface design phase of LISSA, we considered two types of realism. The first one (described in Chapter 2) is visual realism which consists in creating the virtual scene producing the same visual response as the real scene, such as human proportion, character movement, camera viewpoint, etc. The second one is functional 4.3. The LISSA framework 77 realism which consists in creating a virtual scene that transmits the same information as the real scene. Since this first version of LISSA is for PC platforms, all the functions start with mouse and keyboard. According to the concept of functional realism, we apply two ways to interact with the patient. The first one is based on common widgets such as button, checkbox, drop-down list, text box, etc. Figure 4.9 and 4.10 present examples of how to interact with LISSA. Figure 4.9 shows the main menu that contains all available actions, equipments and medicines supported in the game. For example, if player wants to apply the CPR, he has to click on perform CPR, in the action menu. Then, three more steps are required to complete the CPR. First, the player has to select the position where the compression will be applied by clicking on the patient’s body (see Figure 4.10(a)). Second, the player needs to verify which style of hand placement he will use (see Figure 4.10(b)). Once the chest position and hand placement are correct, the player has to apply the chest compression, by pressing the spacebar, and give breath by pressing enter on the keyboard (see Figure 4.10(c)).

Figure 4.9: Action menu of LISSA

The second way of interaction aims at simulating the direct interaction with the patient. In this case, we apply a mouse-over technique which refers to an event graphic user interface that is raised when the user moves or hovers the pointer over a particular area of the graphical user interface. To identify the parts of the patient’s body where the user can interact, we create glyphs over them. When the learner clicks this glyph, the available options according to that body part will appear. In Figure 4.11 we show a victim with these glyphs. This technique improves human-computer interaction by emulating real-world interactions and providing an easy way of use for non-technical people.

4.3.3 Correction Module

The correction module assigns the score to the learner according to the actions he/she has applied in the CPR situation. When a new problem is created, the instructor has to select a set of parameters that determines the correctness of the proposed solution (see Figure 4.7). These parameters are related to some states of the UML state machine (see Section 4.3.2.1). For instance, the instructor can select if patient will breath after open airway and the percentage of success after CPR maneuvers. The instructor can also decide the number of attempts that learner can perform to solve the problem, as well as the playing time which mean that determine when the ambulance or help will 78 Chapter 4. LISSA, a Life Support Simulation Application

Figure 4.10: Some of the CPR procedures that can be performed through the LISSA game

Figure 4.11: Glyphs over the patient to show the user interaction areas arrive and the game will be over. The instructors can also allow the learners to perform as many attempts as he/she wants to solve a problem and with no time restrictions. We denote this option as the practice mode. In addition, the problems can be classified according to their level of difficulty as easy, normal and difficult. This classification is used to determine the obstacle that will be given to the learner to solve the problem. The more difficult the level is, the more obstacle is provided to solve the problem. The obstacle can be actions or medicines that are not within the resuscitation procedure, in 4.3. The LISSA framework 79 order to confuse the player. When a learner enters into the system to solve a problem, he/she has to apply the CPR procedures. Any action will impact on the score which is presented to the user in the form of a performance indicator as shown in Figure 4.12. This indicator is separated into four equal segments that represent poor, fair, good and excellent, from left to right, respectively. The pointer will be in the middle between fair and good segment when the scenario starts. Time usage in the game and any incorrect action causes the minus score moving the pointer from right to left. On the contrary, the correct actions will bring the plus score moving the pointer to the right. This performance indication is presented in real-time to inform on-line the player.

Figure 4.12: The LISSA performance indicator that informs user in real-time

Figure 4.13: LISSA performance report of a problem

During problem execution all the learners actions are recorded in the system database and hence we can create a report for each one of the students. This report is presented 80 Chapter 4. LISSA, a Life Support Simulation Application when the problem finishes and the learner can see his/her progress. This information can also be consulted by the instructor. Figure 4.13 presents an example of a performance report of a problem solving. The report shows four kind of performance data in according with time usage. From top to bottom, the first layer shows the symbol of actions that have been applied, the second layer shows the cardiac cycle of the heart (heartbeat) by number (times per minute) and graph, the third layer shows the respiration signal by graph and finally, the bottom layer shows the score, which is referred by the performance indicator, by graph and percent.

4.4 Results

In this section we present some of the interfaces of the LISSA framework and some implementation details.

4.4.1 LISSA interfaces

LISSA has two main user interfaces, one for the instructor and the other one for the learner. To make the user interaction as simple and efﬁcient as possible, a user cen- tric design was used. In this way, user requirements were considered right from the beginning and included into the whole design cycle. The standard style of interaction such as buttons, list-boxes, checkboxes, textboxes, tabs and scrollbars are used in the graphic user interface components of LISSA similarly to other computational games. The instructor interface mainly allows instructor to manage the problems and tutorials. Figure 4.14 shows two screen-shots of the instructor interface. Note that it is composed of four menus: exercise, tutorial, student and report. The exercise menu (see Figure 4.14(a)) allows instructor to manage problems by adding, editing, removing and assigning them to learners. The tutorial menu (see Figure 4.14(b)) similar to the exercise menu, authorizes instructor to add tutorial materials which can be text, images and videos that can be assigned to the learners. The student menu presents the list of learners and the classroom in order to view their past performance and assign them the problems. The learner interface (also denoted as student workbook) mainly allows learner to solve the problems in the simulation, review an exposition case (related with a tutorial) and review the past performance (the history proﬁle). Figure 4.15 shows four screen- shots of the learner interfaces. Figure 4.15(a) corresponds to the home interface which presents the list of problems assigned to the learner by the instructor. In this list there is also information about the number of previous attempts to solve the problem, time spent in the problem and links to the comments introduced by the teacher. To solve a problem the learner has to select it from the list. Then, the description of the emergency situation appears and the problem starts. Figure 4.15(b) presents the tutorial menu within learner interface, this presents the e-learning materials that instructors keep in the database. Figure 4.15(c) shows the exercise that has been assigned to the learner by the instructor. The exercise interface is presented in Figure 4.15(d). 4.4. Results 81

Figure 4.16 shows the sequence of actions that have to be performed to solve a CPR emergency where no medical devices are considered. The scene always starts with the emergency situation, when a patient falls down for different reasons in various places, such as football field, swimming pool, bus stop, in the hospital, etc. The learner performs as a helper trying to revive the patient in the limited time. After finishing each mission the score will appear and the learner can choose the next mission for progress. The resuscitation includes heart attack or stroke patient, or any scenario where breathing or heartbeat have been compromised. In some cases, the automated external defibrillator (AED) is available. From left to right, Figure 4.16, shows main scenes of the problem: (a) information of the patient is presented to the learner, (b) initial state with patient on the floor where learner has to apply the action by clicking on the check respond button then (c) the responsive check is performed, from this state if the patient has responsiveness the system will turn to state (d) where observation is required. On the contrary, if the patient has no responsiveness next procedure has to be (e) that is ask for any help from the bystander. Then (f) the patient airway needs to be opened and followed by check breath in (g). From this state, in case that the patient is breathing, helper has to place him into recovery position as presented in (h). If the patient has no sign of breathing, (i) emergency call is required (i). Then (j) the patient cloth (top) needs to be removed before CPR procedure is applied. In (k) to perform CPR the correct chest position where the compression will be applied needs to be specified, as well as, the hand placement as shown in (l). Next follow (m and n) where 30 chest compressions and 2 breath giving are applied, respectively. The CPR (30:2) has to go on until (o) the help or emergency unit arrives which means that the game is over. Finally, (p) the performance report will be presented. A systematic evaluation focused on teaching and learning effectiveness of LISSA is presented in Chapter 7.

4.4.2 Implementation details

The LISSA project follows a client-server architecture, where the client (main application) requests all the needed information to the server. This information is related to the user, problems and activities. The client part has been implemented using Unity 3D. This software is a game engine that shows the user interface: problem deﬁnitions and virtual representations of them (patient, helper and the environment). Unity 3D allows the programmer to write the code in C#, Javascript or BooScript, but the most used and the one chosen for the development of LISSA is C#. The second part (server part) is implemented using an Apache client. Apache will process all requests received from the Unity3D client and will return a response with all requested data. This information is stored in a MySQL database which is installed also in the server, so Apache will have local access to it. 82 Chapter 4. LISSA, a Life Support Simulation Application

Figure 4.14: LISSA graphic user interface for the instructor 4.4. Results 83

Figure 4.15: LISSA graphic user interface for the learner 84 Chapter 4. LISSA, a Life Support Simulation Application

Figure 4.15 LISSA graphic user interface for the learner (cont.) 4.4. Results 85 Problem solving Figure 4.16: 86 Chapter 4. LISSA, a Life Support Simulation Application

4.5 Conclusion

Cardiopulmonary respiration (CPR) is a crucial part of basic life support techniques. Different strategies have been proposed to teach the CPR procedures. In this chapter, we have proposed LISSA, a serious game for teaching and learning CPR. The game turns around an emergency case which has been modelled as a state machine that reproduces a CPR flowchart. 3D graphical representations and animations have been proposed to model the states and the transitions between them. LISSA has been designed to support two type of users, instructors and learners, and integrates different e-learning functionalities to allow the definition of problems of different levels of difficulty, the assessment of the learners or the consultation of actions performed by learners. The first version of LISSA, based on PC with mouse and keyboard, has been evaluated positively by a group of experts from the infirmary faculty of University of Girona from Spain and technology and environment of Prince of Songkla University from Thailand. The evaluation of our platform, teaching and learning effectiveness, is presented in Chapter7. CHAPTER 5 Visual realism design for LISSA

5.1 Introduction

Researchers in many areas consider game design, with regard to aesthetic context, as a key element in developing successful games [117, 162]. Generally, game players tend to value good graphics, animation and sound. These factors provide sensory proof of the reality of a game, supporting and enhancing the impact of the whole game experience. Rhyne et al. [181] also suggests that the higher the level of reality or fantasy is, the more entertaining a game becomes. A realistic scenario increases the player satisfaction and sound effects help to establish the ambience or atmosphere of a game by increasing realism. Therefore, graphics and scenario play the key roles in the visualization aspect. Developing a serious game with regard to the realism has to take into account both the cutting edge technology and players perception. In this chapter, we present a study of visual realism design. As a case study we are going to consider LISSA, the LIfe Support Simulation Application we have developed to teach and learn cardiopulmonary resuscitation (CPR) (see Chapter 4). LISSA provides a 3D computer simulation of emergency scenarios where a patient needs CPR application. We aim at evaluating different visual elements, photorealistic against non- photorealistic rendering as well as different camera viewpoints that impact on visual realism.

5.2 Background

Visual realism in computer-simulated environments is considered as Virtual Reality (VR), a term referring to the simulation of physical presence in places in the real world, as well as in imaginary worlds. Most current serious games for learning provide primarily visual experiences of the real simulated objects and present them in visual environments with different level of realism. In order to optimize the level of realism in a game, ﬁve visual realism elements (form, texture, illumination, movement and viewpoint) have to be considered. Each element can be created from realistic to abstract and the properties of one element affect to the other one. For example, super distortion proportion has funny look which is good at increasing children0s motivation but it can not be set to some posture so this form may not be suitable for games which considers the correct human factors. The visual realism elements are closely related with the 3D production process as it is illustrated in Figure 5.1. More details are described in Section 2.7. 88 Chapter 5. Visual realism design for LISSA

Figure 5.1: Relationship between visual realism elements and the 3D production process

Since a game is an interactive virtual environment, it is necessary to balance the level of realism and the interactive quality of the virtual environment. For example, it is not only the form that we need to consider in the modelling phase but we also need to control the interaction fidelity by minimizing the number of polygons in a scene in order to maintain the smooth real-time rendering while playing. Interaction fidelity refers to the degree at which the simulator technology represents the real world scene to the user. A well balance between level of realism and interactive quality will obtain a good aesthetic visualization and smooth play in the predefined environment. To evaluate all visual realism elements in a real case we considered as a case study the visual production of LISSA.

5.3 Visual realism within LISSA

To study visual realism in the context of LISSA, we need to consider the main visual elements involved in the production of the game, i.e., form, textures, Illumination, movement and viewpoint (see Figure 5.1). In our game, the main scene is the CPR scenario which reproduces the emergency situation that requires the CPR procedures. Therefore, we are going to analyse the visual elements by considering a CPR scene.

5.3.1 Form

The first element to be considered is the form which defines the different objects that compose our scene. We have to take into account two main aspects, the first one is the proportion of the objects and the second one is the number of polygons used to create the form. Let’s consider each one.

Proportion: the main elements of our CPR scenario are the patient and the helper. • 5.3. Visual realism within LISSA 89

Since we want to reproduce them with maximum realism with respect to real humans our objects need to keep real proportions. Moreover, a real proportion is required in order to maintain the real kinetic and human factors during the emergency scenario (see Chapter6 for more details). To satisfy our requirements we modelled the characters as an average person, generally seven and a half heads tall (including the head) which is the proportion of humanoid form (see Figure 5.2).

Figure 5.2: Human character proportion in LISSA

To model the equipments and other 3D objects in the scenes, the real proportion is also maintained.

Number of polygons: since LISSA is a real-time computer-based application, the • size of all related ﬁles need to be as small as possible to minimize the rendering time. In the modelling phase, this consideration is related with the number of polygons used to model the scene. In our case, the number of polygons is limited by Unity3D, the game engine used for the game development. Unity3D allows up to 60,000 polygons per object. Polygons of LISSA objects are between 5,000 to 30,000 depending on their structure. To ensure real-time interaction and preserve image realism we created scenes where the number of polygons for one real-time scene, including characters, equipments and environment, were between 40,000 to 200,000. Figure 5.3 shows the polygons used to model an ambulance which are 91,579 polygons in total. To minimize the number of polygons, modellers needed to maintain the visual realism as similar as a high-polygon model. The reduction of polygons were performed until modeller found a reasonable compromise between visual quality and performance for real-time interactive use where usability is more important than visual perfection. 90 Chapter 5. Visual realism design for LISSA

Figure 5.3: Number of polygons in ambulance scene

Figure 5.4 presents an actor of the game modelled with a low and a high number of polygons, see Figure 5.4(a) and (b), respectively. The two models were textured with the textures presented in Figure 5.4(c). The final models are presented in Figure 5.4(d) and (e). We can see that the results are the same even though the number of polygons is very different (3,996 for low polygon model and 251,120 for high polygon model). In addition, some techniques were used in order to control the number of polygons in a rendered scene such as the application of textures polygons instead of full geometrical models. This can be applied when an an object is normally viewed only from a few sides or if the object is mostly flat such as a bookshelves or poster on the wall. Furthermore, some unimportant and hidden objects were identified, simplified and completely eliminated, e.g., if there is a lab-top computer in the scene and the player does not request to open it the keyboard is not modelled.

5.3.2 Textures

The second visual element that have to be considered are textures. Texture in visual arts is referred to apparent surface which present its physical properties such as fur, wood, sand, smooth, metal, glass, leather, etc. To maintain the smoothness while playing, the textures file size, which is related to type of texture (vector or bitmap) and the texture file format, need to be considered. In order to overcome the limitations of per-vertex surface, which requires a high-level of tessellation to achieve a smooth look, we use bitmap images (multi-texture) which contain color information and are usually projected onto the triangles of a mesh. In this way, even we use the bitmap- based texture, that generally has a bigger file size than the vector-based texture, the high polygon model is not necessary. It can be mapped onto the low-polygon model 5.3. Visual realism within LISSA 91

Figure 5.4: Difference between low-polygon model and high polygon model and presents more realistic visualizations. When creating a character model, the high polygon model is ﬁrst scrubbed in order to keep the details of folds and wrinkles on clothes then the topology is changed by turning the sculpted object into a low polygon mesh. The details within the high polygon model are unwrapped and turned to a realistic texture. Most of the texture maps used within LISSA are jpg format which can be lossy compressed. On the other hand to represent the transparent objects we use tiff format since alpha channel is supported. In order to provide material details as well as the pre-deﬁned lighting onto the textures, we apply an image-based lighitng technique. In this way, the bitmap graphics are composed of diffuse and ambient occlusion maps. Figure 5.5 and 5.6 show examples of textures used in our game. 92 Chapter 5. Visual realism design for LISSA

Figure 5.5: Some of the texture maps used for a LISSA character

Figure 5.6: Some of the textures used for the automatic deﬁbrillator machines 5.3. Visual realism within LISSA 93

5.3.3 Illumination

The third visual element to be considered is illumination. All the LISSA scenes are daytime scenes since we consider that they present more clear situations. There are also some scenes in an ambulance or in hospital. Our lighting concept is to provide the light enough to see all objects and actions. To light the scenes and the objects, we used the basic lighting functionalities of the Unity3D game engine [217]. Unity3D has four different types of light: directional light, a light placed infinitely far away and affects everything in the scene; point light, a light that shines equally in all directions from its location and affecting all objects within its range; spot light, a light that shines everywhere within a cone defined by spot angle and range and only objects within this region are affected by this light; and area light, a light that shines in all directions to one side of a rectangular area of a plane. Area lights are only available during light-map baking and have no effect on objects at runtime. In order to initiate the realistic lighting, we have to apply these four light sources provided by Unity3D considering the place (exterior or interior) and the time (day or night time). Furthermore, in order to minimize the rendering time, we used image based lighting techniques which do not require render computation. In LISSA, our textures were embedded (baked) with indirect illumination such as ambient occlusion, some caus- tics (refractive, reflective) and post-processing functions (saturation, contrast). These baked textures present how light reflects from the model’s surface. In this way, some illumination effects such as shade and shadow can appear onto the texture maps without any light source (see Figure 5.5).

5.3.4 Movement

The fourth element to be considered is movement. There are two kinds of movement in 3D scenes, object and camera movements. All the movements in LISSA are related to the CPR actions. Some of them are change patient position (supine or lateral), open airway, give breath and compress patient0s chest. All these actions are performed by the helper with a consequence on the patient. During the animation process, the main issue to be considered is the collision between helper and patient. We have to avoid overlapping. In Figure 5.7, we illustrate two examples of these actions. Note that when the helper hands touch or hold part of patient body it is necessary to place the hands in the correct position. Another requirement of the animation is the correctness of the movements, since LISSA is deﬁned as a serious game for learning, the posture and movement of characters have to be in the correct way. Apart from using a motion tracking device such as motion capture that provides high accurate movements, the rotoscoping technique by animating the model over a video footage can be also used. In LISSA, to animate the models we used rotoscoping. 94 Chapter 5. Visual realism design for LISSA

Figure 5.7: Example of high precision animation while two objects have touched

5.3.5 Viewpoint

The last element to be considered is viewpoint. With regard to visual realism, user viewpoint can be separated into two study contexts which are camera viewpoint and rendering effect. There is a variety of camera view (shot, angles and viewpoints) as illustrated in Fig- ure 5.8, as well as, a variety of rendering effect as illustrated in Figure 5.9. Therefore, it is difﬁcult to determine by default which are the viewpoints and rendering effects that better satisfy user preferences. For instance, even if the ﬁrst person view should be the most realistic view, it may not be the preferred one by our target user due to it is lack of ability to see the avatar body [163].

Figure 5.8: Viewpoint 5.4. Visual realism and user preferences 95

To tackle this problem, we decided to carry out different experiments to determine which are the preferences of the users with respect to camera viewpoints and rendering effects. All the details of these experiments are presented in next section.

5.4 Visual realism and user preferences

Some visual realism elements, such as photorealistic or non-photorealistic rendering effect or camera viewpoint, are elements that can vary according to the user preferences. Not all the users desire the same effects. In order to evaluate the preferences with respect to these elements, we designed three different experiments. The first experiment aims at comparing the user preferences with respect to photorealistic (PR) and non-photorealistic (NPR) rendering. Since there are many NPR effects that can be used, to carry out this test we decided to choose only one NPR style in order to compare with the PR one. To determine the NPR effect we defined a set of NPR scenes with different effects (see Figure 5.9). Then, we analysed the images and selected their preferred one. All the members decided that the best one for the test is the one represented in Figure 5.9 (h). The second experiment has been designed to compared between 1st and 3rd person viewpoint. To carry out this experiment participants have to play LISSA with 1st and 3rd view. To see the differences between these viewpoints, in Figure 5.10 we show different scenes of the game rendered in both modes. The third experiment has been designed to evaluate the user perception considering both a pre-determined photorealistic (PR) rendering style and the 3rd person viewpoint. This predefined elements (PR and 3rd viewpoint) has been determined by the development team since we found that the 1st person viewpoint lead to confusion while the helper perform CPR chest compressions and give breath and the photorealistic is a default rendering style that most seen in the game market. Once the three experiments have been defined, a group of 22 volunteers with computer experience.has been selected. The same protocol was followed with each experiment. Our evaluation was based on a questionnaire. First, the participant completed the pre-experiment questionnaire. Second, the participant was told the goal of the experiment. Third, the participant was given the chance to practice playing the game in four demo modes: (i) PR with 1st viewpoint, (ii) PR with 3rd viewpoint, (iii) NPR with 1st viewpoint and (iv) NPR with 3rd viewpoint (see Figure 5.10 and 5.11), in randomized order for no more than three minutes. Fourth, the participant played the game with a pre-determined user interface, which is PR with 3rd viewpoint, on a pre-determined level. Finally, the participant answered the interface experience questionnaire for that user interface. The questionnaire has two main parts. The first part is a pre-experiment questionnaire, which gathered the participant’s characteristics including gender (male/female), handedness (right/ left/ambidextrous) and age (from <20 to >60 years old). The participants were also asked to rate their expertise with computer and game. Participants rated their expertise by selecting the frequency of using computer and playing game (from <1 to >20 hours per week). The first column of Table 5.1 collects the formu- 96 Chapter 5. Visual realism design for LISSA

Figure 5.9: Examples of NPR rendering for pre-selection 5.4. Visual realism and user preferences 97

Figure 5.10: Comparison of PR and NPR to LISSA

Figure 5.11: Comparison of 1st and 3rd person viewpoint in different actions 98 Chapter 5. Visual realism design for LISSA lated questions. The second part is an interface experience questionnaire composed of three sections. The first section was related to the first experiment. The volunteer played the game in two different modes, photorealistic (PR) and pre-determined non- photorealistic (NPR) style. Figure 5.10 shows the same scene rendered with different rendering styles. The second section is related to the second experiment which respect to player viewpoint, where the participants were allowed to play LISSA with two different viewpoints (1st and 3rd) and marked their preferences. The last section is related to the third experiment and evaluates the overall user perception of our interface. Pre- defined rendering style (PR) and viewpoint (3rd view) were set up. The participants were asked to rate how much they agree to the formulated question. We used Likert scale, a scale of psychometric responses widely used in survey research, that ranges from strongly disagree, disagree, neutral, agree and strongly agree [220].

5.5 Results

The results of the study from the 22 participants who are between 21 to 60 years old are presented in this section. From the ﬁrst questionnaire (see table 5.1) we can see that most of the participants (81.81%) use computer more than 20 hours per week and 18% do not play games.

Issue Details Gender 8 men, 14 women Handedness 19 right, 1 left and 2 ambidextrous Age (years old) 20 (21-40) and 2 (41-60) Frequency of computer using (hour per week) 4 (1-10H) and 18 (>20H) Frequency of playing game (hour per week) 4 (no), 6 (<1H), 8 (1-10H), 1 (11-20H) and 3 (>20H)

Table 5.1: Characteristics of sample for the questionnaire

The results of the ﬁrst experiment are presented in Table 5.2. In this case our purpose was to compare photorealistic rendering (PR) and non-photorealistic rendering (NPR). Five questions about scene understanding (Q1), attractiveness (Q2), motivation (Q3), satisfaction (Q4) and suitability (appropriate with LISSA) (Q5) have been asked. The answers to all the questions are unanimous that photorealistic rendering is preferred to non-photorealistic rendering. The results of the second experiment are presented in Table 5.3. With respect to the comparison of 1st and 3rd person view, four questions about realism (Q6), comfortability (Q7), satisfaction (Q8) and suitability (Q9) have been asked. We can see that 3rd person viewpoint gained an overwhelming score (0.8<µ<0.9) except for the realism question which although received positive rate ( (µ=0.9) but the variance is also high (σ=0.8). Finally, the results of the third experiment are presented in Table 5.4. With respect to the visual perception of LISSA, seven questions, motivated by realism (Q10), level of realism (Q11), joy (Q12), ease of use (Q13), comfortability (Q14), predictability (Q15) 5.6. Conclusion 99

Questions -1 µ σ 1 Q1. Which one was easier to understand? NPR 1.0 ± 0.2 PR Q2. Which one was more attractive? NPR 1.0 ± 0.0 PR Q3. Which one motivated you to learn more? NPR 1.0 ± 0.0 PR Q4. In overall, which one satisﬁed you more? NPR 1.0 ± 0.0 PR Q5. Which one is more appropriate with LISSA? NPR 1.0 ± 0.0 PR ± Table 5.2: Comparison results of photorealistic (PR) and non-photorealistic rendering (NPR)

Questions -1 µ σ 1 Q6. Which viewpoint made you feel more real? 1st 0.6 ± 0.8 3rd Q7. Which viewpoint was more comfortable to use? 1st 0.9 ± 0.4 3rd Q8. For overall satisfaction, which one satisfy you more? 1st 0.8 ± 0.6 3rd Q9. Which one is more appropriate with LISSA? 1st 0.8 ± 0.6 3rd ± Table 5.3: Comparison results of 1st and 3rd person viewpoint and effectiveness (Q16), have been asked. All answers are positive (3.7<µ<4.5). Note that, there are two issues, which are ease of use and predictability, that even if they got the positive score the variance is quite high (σ=0.9 and 1.0 respectively). The average score (Q10-Q16) for non-gamer participants is good (3.82 out of 5) as well as for elder participants (4.07 out of 5). In a posterior interview with elder users they remarked that an introductory tutorial and a guide for learning how to play the game would be welcomed. Figure 5.12 shows evaluation results of all characteristics with regard to the questionnaires as detailed previously. There are three different color lines presented on the spider web graph which are average results (in red) and error polylines (green and blue) of standard deviation. The results show that willingness of learning can be enhanced by visual realism. Photorealistic rendering and 3rd person viewpoint was well rated. Focusing on the user perception of a pre-determined LISSA interface, photorealistic rendering and 3rd person view, participants have positive feeling with the degree of visual realism, enjoyment, ease of use and comfortability while playing the game. Participants also agree that the intended actions were accurately carried out on screen when using LISSA interface. In the learning context, participants feel that learning with LISSA, their CPR performance can be improved signiﬁcantly.

5.6 Conclusion

Visual realism is one of the major issues in games, in order to optimize the system performance together with user satisfaction. In this chapter, we have presented the process of optimizing visual realism within LISSA, our serious game. We have analysed the ﬁve visual realism components (form, texture, illumination, movement and viewpoint) of the game production process. Each element has been analysed in the context 100 Chapter 5. Visual realism design for LISSA

Figure 5.12: Evaluation of average results (in red); error polylines (green and blue) at one standard deviation. R, VP and L denote questions related to rendering mode, view point and the effectiveness of LISSA, respectively. 5.6. Conclusion 101

Questions 1 µ σ 5 Q10. Visual realism enhance the will- Strongly Disagree 4.5 ± 0.5 Strongly Agree ingness of learning ± Q11. The degree of visual realism in Strongly Disagree 3.8 0.6 Strongly Agree LISSA is high enough ± Q12. Using this interface was enjoy- Strongly Disagree 3.8 0.7 Strongly Agree able ± Q13. Learning this interface was easy Strongly Disagree 3.7 1.0 Strongly Agree Q14. This interface was comfortable Strongly Disagree 3.7 ± 0.6 Strongly Agree to use ± Q15. My intended actions were accu- Strongly Disagree 3.7 0.9 Strongly Agree rately carried out on screen when us- ± ing this interface Q16. I feel like more practice time Strongly Disagree 4.2 0.5 Strongly Agree with this interface would have made ± a signiﬁcant difference to my performance

Table 5.4: Visual realism results of LISSA of LISSA. In the modelling phase, we have used low-polygon models with the real proportion to maintain the real kinetic and human factors. In the texturing phase, we have used bitmap-based textures with image based lighting techniques such as ambient occlusion which was embedded onto the texture maps in order to minimize the render resources. In the animation process, the movement is related to the CPR actions. Since LISSA is a formal CPR teaching/learning tool, the correct human movement is preserved. In the lighting phase, basic illumination provided by Unity3D game engine has been used. Our lighting concept is intended to provide enough lighting to see all objects and actions. Finally, in the rendering phase, we have carried out a study to determine the preferences of the user with respect to photorealistic and non-photorealisitc rendering effects and camera viewpoints. The evaluation results stated that photorealistic rendering with 3rd person viewpoint are well rated and participants feel positive with the visual realism of LISSA. These results confirm both that realism plays a key role in teaching by simulation applications, as well as the observations made by Pepperell [163] about the difficulties of the realistic depiction of peripheral vision in first person view including the limited visibility of the viewer body.

CHAPTER 6 Kinect-based system for LISSA

6.1 Introduction

Simulating social scenarios in 3D virtual environments, such as interactive learning frameworks or games, requires extensive interaction design and tools. In order to control an avatar, many strategies can be applied including functional, verbal and gestural communication. Functional communication normally uses interaction tools like mouse and keyboard dealing with screen functionalities such as buttons or hyperlinks. To improve this interaction, verbal and gestural communication can be used. These requires supporting from special equipments. In the case of serious games that require a high level of realism, the use of communication strategies are necessary in order to enhance the physical realism. Physical realism is related to player interaction tools which could be enhanced by using motion tracking systems or haptic devices, a tactile feedback equipment that takes advantage of the sense of touch by applying forces, vibrations, or motions of the user. For example, in some touch-based system such as surgical simulator, the haptic device is not only used to improve the physical realism, but also to provide important diagnostic information through the sense of touch [242]. Also human movements tracking systems can be applied to enhance the physical interaction. For example, a tele-rehabilitation system can encompass a web-cam for ﬁnger hand movement tracking of patients during local and remote therapy sessions [125]. Due to the importance of user interaction on the context of serious games, we aim at studying some of the techniques that have been proposed and analyse how we can use it to improve our serious game. The ﬁrst LISSA prototype, presented in Chapter 4, is based on basic interaction techniques provided by mouse and keyboard interaction. In the context of cardiopulmonary resuscitation learning, these techniques are not as effective as manikin-based learning where the user can interact directly with the manikin. In this chapter, we analyse how to improve the physical realism of LISSA considering the key parameters of the CPR procedure and how they can be solved by a human tracking system. We propose a Kinect-based system to reproduce the CPR procedures with a high level of realism.

6.2 Background

Interaction technology has different paradigms for establishing communication between humans and computers. Both hardware and software interfaces are included. Traditionally, mouse and keyboard have been used; newer means of interaction include 104 Chapter 6. Kinect-based system for LISSA virtual reality (using head-mounted displays), computer monitors, haptic or pseudo- haptic devices such as gloves or pens, or tracking devices. Webcams and web applications are also common. Users can affect virtual objects in real-time using a variety of senses (vision, hearing and touching). In this section, we review some of the most representative human motion tracking tools and techniques. We will give particular attention to Kinect since it is the basis of our proposal.

6.2.1 Human motion tracking system

Performance capture is required by many fields of use such as animation movie, virtual reality, rehabilitation, ergonomic study and many more [240]. The tracking system can be separated into many different aspects such as: wired and wireless; optical and non-optical based; and real-time and non-realtime. In this thesis, we will focus only on the interaction tools for wireless and real-time human motion tracking system. Real-time human movement tracking systems are expected to generate real-time data that dynamically represents the pose changes of a human body (or a part of it), based on well developed motion sensor technologies. Different motion capture technologies have different characteristics in terms of use and overhead, differences which affect the overall system value in significant ways. In practice, the advantages and disadvantages of each approach depend on the applied domain. For example, marker- based optical motion capture systems can provide a high precision data but require the subject to wear special reflective or lighted markers suit and have to set up the system in specific area which takes a lot of time to calibrate the device. Therefore, this system is not suitable for game-based consumers but for professional that requires high accuracy information. Generally, human motion tracking system can be separated into optical and non- optical based systems. Optical-based motion tracking utilizes optical sensors to capture the 3D position • of a subject between one or more calibrated cameras in order to provide overlapping projections. Multi cameras are normally applied to improve accuracy in position estimation. A huge benefit of optical systems is their capability to track multiple objects at a time, limited by only the resolution and field of view of the camera and the available processing time. To identify the subject, two approaches are used;

– Marker based system which makes use of special markers attached to the subject. These markers are designed to be easily identiﬁable by image processing software. Typically, the markers are either highly reﬂective balls or else small bright lights that stand out from the background and are easy to identify by computer vision algorithms. – Marker free system which uses advanced computer vision technologies to identify and track subjects without the need for any special suits or markers. In order to do that advanced image processing algorithms, such as 6.2. Background 105

image segmentation or depth processing are required. However, extraction of depth information is a challenging task when only color images are available — e.g. for a still image it cannot be solved. Modern marker free motion tracking devices, such as the Kinect (Section 6.2.2), are thus equipped with a depth sensor that provides accurate distance information from the camera.

Figure 6.1: Optical-based motion tracking system with markers (left) and marker-free (right)

Non-optical-based motion tracking uses inertial sensors, such as gyroscopes and • accelerometers which are suitable for capturing fast movements [19]. The motion data of the inertial sensors can be transmitted wirelessly for further processing or visualisation. Inertial sensors can be of high sensitivity, large capture areas, sampled at high rates, require only little energy and do not suffer from magnetic interferences. On the other hand, accelerometers and gyroscopes suffer from the so-called integration drift [184, 244]. They measure the derivatives of the 3D position from which the position is obtained via integration over time, also accumulating measurement noise. However, with the evolution of the accuracy of accelerometers, the accumulated noise is still low when the device is used only for a short period of time. A more important drawback of accelerometers and gyroscopes is that they cannot track multiple targets. Thus, for a system that needs to evaluate e.g. poses or gestures of the user, multiple devices should be used, one for each joint.

Consumer gaming solutions were never designed to be very accurate and cannot scale to full 3D capture due to inherent technical limitations with infrared sensors. Marker-based systems are inherently difﬁcult to use since the area of use is limited

6.2.2 Kinect

Microsoft Kinect is a webcam-style add-on peripheral designed to support natural user interface such as gesture recognition or spoken commands. Originally intended for the Xbox 360 game console, launched in November 2010. In addition to its color camera (RGB camera), it is also equipped with an infrared sensor and camera providing depth 106 Chapter 6. Kinect-based system for LISSA information (see Figure 6.2), turning the Kinect into a low-cost ($89.99 US, February 2013), real-time full body 3D motion capture device. According to the documentation, two skeletons, and up to 6 people within its ﬁeld of view can be detected by Kinect. For a single skeleton, 20 joints can be used in stand posture and 10 joints while sit. By analysing the gestures and poses of the user, Kinect can provide basic interactions similar to mouse, keyboard and touch interactions (i.e. selecting buttons, zooming and panning around a surface) [139].

Figure 6.2: Microsoft Kinect

A growing number of empirical studies have evaluated the effectiveness of Kinect technologies [139, 160], as well as its influence on serious games for health [33, 35, 65, 195]. Several health serious games based on Kinect peripheral have been reported so far. Kinect based serious games for health focused on exercise and wellness. Some examples are EA Sports Active [56], Fishing Cactus [74], Your Shape: Fitness Evolved [215] and Zumba Fitness [245]. Later, from 2011 until now, Kinect is used in more specific health games such as Kinetix Academy [114], a set of games for motor development with autism or Schnauer et al. [191], a chronic pain rehabilitation game and Voracy Fish [81], a serious game for the upper limb rehabilitation. Kinect provides two levels of data for application developers. The first one is the captured raw data, namely the color and depth images. These can be inputs of computer vision algorithms to recognize and track objects (e.g. human joints). Addi- tionally, general-purpose skeleton tracking algorithms were developed by experts and packed into middle-wares such the Microsoft Kinect SDK or NITE [140, 168]. The benefit of using the raw images is that it allows to apply application specific algorithms, for example to track a marker in the 3D space with high precision. A good example for this is VREM [195], which tracks a colored bracelet to obtain CPR compression depth with 2mm precision. Skeleton tracking on the other hand is general-purpose and thus, it has lower precision and robustness, but it is more straightforward to seamlessly integrate into an existing games environment, such as LISSA. Therefore, in this work we chose to build our application on skeleton tracking as an interactive tool in CPR learning application, and we will consider the use of computer vision algorithms as future work. To better understand our proposal, a brief description of LISSA is given in next section. 6.3. Human motion tracking tool for CPR 107

6.3 Human motion tracking tool for CPR

To improve the interaction of our CPR learning framework, we need to evaluate the main actions that need to be performed when solving a CPR emergency. If we analyse the CPR protocol [63], we can see that the continuous chest compression is the key procedure. This procedure aims at maintaining blood circulation through the heart and brain by manual pumping [210]. Therefore, we need to focus on the parameters related with chest compression. Five criteria need to be considered to correctly perform chest compression. These are: (i) hand position on the chest, at the lower margin of patient rib cage; (ii) hand placement, by criss-crossing the hands with the dominant hand on the hand already placed on the chest; (iii) compression depth, up to 2 inches or 4-5 centimeters; (iv) compression rate, about 100-120 pushes per minute; and (v) arm position, creating a 90-degree angle with victims body (see Figure 6.3).

Figure 6.3: Correct CPR chest compression sequence presented in LISSA

The automatic evaluation of these CPR criteria requires the acquisition of certain information from the user. In Table 6.1, we collect for each criteria, represented in the second to fifth columns, the required information. For each criteria, we report the part of the body to be evaluated and the type of required information, where absolute means that we have to obtain the given parameter in absolute measures, while relative denotes that it has to be known only with respect to another object. For example, compression depth can be defined as the hand movement in the direction perpendicular to the chest in millimeters, while correct chest position is given by the user0s hand with respect to the rib cage, regardless the exact distance. To evaluate chest compressions made by a trainee according to the criteria listed above, the human pose and motion has to be tracked using a motion tracking system. Each of these systems has its own limitations and advantages which have to be considered when designing a CPR teaching system. Our goal is to develop a cheap and compact system that can be widely used in everyday education. Table 6.2 shows the characteristics of different motion capture systems that fit our purposes, taken from the survey of Zhou et al [244]. Additionally, as Table 6.1 shows, formal evaluation of 108 Chapter 6. Kinect-based system for LISSA

Criteria Hand Hand Compression Compression Arm position placement depth rate position Joint hand hands and ﬁngers hand hand arm joints

Required relative position relative absolute movement relative position information w.r.t. rib cage position position direction w.r.t. each other

Table 6.1: Required information with regard to CPR chest compression criteria (where absolute means that we have to obtain the given parameter in absolute measures and relative denotes that it has to be known only with respect to another object.

Systems Accuracy Compactness Cost Drawbacks Example Tools Inertial High High Low Drifts Accelerometer, Wii, iPhone Marker High Low Medium Occlusion Motion capture system Marker-free High High Low Occlusion Kinect

Table 6.2: Comparison of different motion tracking performance with respect to [244], Note: last column presents only devices compatible with CPR actions in our platform. certain criteria for CPR compression (e.g. arm position or compression depth) requires 3D position of human joints with high precision, which, regarding our intentions on making a cheap and compact system, could be best achieved by marker-free tracking. Inertial systems satisfy the requirements of low-cost and compactness, however, absolute 3D position is very unstable due to the integration drift problem, bias from error position [23]. Amongst marker-free tracking platforms, Kinect is one of the best options by providing 3D position information in real-time, packed in a compact and low-cost device. Therefore, in our work, we use Kinect as the main sensor for the evaluation of CPR compressions.

6.4 Kinect-based system for LISSA

To integrate the Kinect technology in our game, we analysed the main factors of the CPR procedure and how they can be supported by the device. First of all, we focus on the main parameters, which are the correct arm position and the compression rate checking. Then, we focused on other parameters which unfortunately are not supported by Kinect basic functionality.

6.4.1 Arm position

One of the key parameters when applying CPR procedure is the position of the arms. The correct arms posture has to be straight and somewhat rigid since the body has to be directly over the hands. In Figure 6.4, we present two different situations, (a) illustrate the correct and (b) the incorrect positions. Note that one can detect if the position is correct by simply measuring the angle between the hand, elbow and shoulder. To 6.4. Kinect-based system for LISSA 109 detect the correct arm position we only need to deﬁne a set of geometrical constraints on the joint positions returned by the skeleton tracking middle-ware.

Figure 6.4: Example of correct and incorrect arm position

6.4.2 Counting compressions

Since there is a certain compression number that we need to follow regarding to patient maturity, e.g., 30 times for an adult and 15 times for a children, it is necessary to count the number of compressions per second. To count the compressions number, instead of using a bracelet to track the arm position as in [195], we track the hand position returned by the skeleton tracking middle-ware and follow its movement. We consider that a compression is triggered when the direction of this movement changes from down to up. As a side effect, this also adds a minor (around 1 second) delay to compression detection.

6.4.3 Other CPR factors

The remaining three criteria are detection of hand position on patient0s chest, hand placement and compression depth, that can not be controlled by Kinect. To control them we deﬁned a set of new interfaces, presented in Figure 6.5. The ﬁrst one (see Figure 6.5(a)) is used to determine the patient chest position where the compression will be applied by clicking on the patient0s body and the second one (see Figure 6.5(b)) is used to verify which style of hand placement that learner will use. About compression depth, our LISSA current version can not still achieve it. We propose to improve this criteria as a future work.

6.4.4 Implementation details

We integrate Kinect into LISSA focusing on two CPR criteria, the correct arm position and the chest compression rate. In order to track the user skeleton in the game control module, we integrate Kinect into LISSA platform using the ofﬁcial OpenNI wrapper for 110 Chapter 6. Kinect-based system for LISSA

Figure 6.5: Some of the CPR criteria that can be performed through the LISSA game

Unity 3D and NITE as a middle-ware. In Figure 6.6 we illustrate the main modules that compose LISSA version. As it can be seen the main modiﬁcation is the introduction of the Kinect and skeleton tracking modules.

Figure 6.6: The integration of Kinect into LISSA system

The main advantage of this approach is its easier calibration together with easy integration into an existing game environment. The main limitation is the increase of positioning error (by one order of magnitude) and the decrease of robustness (see Section 6.6). We want to mention that by following the hand, obtaining the compression depth would be possible in theory, however in practice, the detection error has the same order of magnitude as the regular compression depth.

6.5 Results

In this section we present the experiments that have been carried out to test the proposed approach. The integration of Kinect in LISSA permits us controlling new CPR parameters, such as (i) average compressions per minute; (ii) total time for chest compressions; (iii) corrected chest compression (with regard to the correct arm position) 6.5. Results 111 and (iv) average time between compressions. To present this information to the user we have introduced new icons in the interface. Please note that, this is a prototype system and the interface is just minimalist, designed only for the experiment. In Figure 6.7(a) we illustrate the whole screen where the black area on the top left displays average compressions per minute, total time, number of correct compressions and average time between compressions. A more detailed view is given in Figure 6.7(b). In Figure 6.7(c) we show how the arm position is presented. Note that a human symbol appears on the middle-top of the screen, where a green symbol means that the user performs CPR-chest compression correctly and a red symbol means that user performance is incorrect.

Figure 6.7: (a) Screenshot of LISSA based on Kinect connection system; (b) the CPR- chest compression rate information and (c) the arm position tracking symbol - green means correct position and red means incorrect position

Our first experiment has been designed to compare the performance of LISSA with respect to iCPR, an iPhone application working by using the built-in accelerometer to display the rate and count of compressions, and Laderal PC Skillreporting System, a computer based CPR training system connected with a special manikin (Resusci Anne). We considered the chest compression rate (times per minute) and the ability to detect the correct arm position. To carry out the experiment an expert from the infirmary faculty of our university supervised all the actions. In Figure 6.8, we show the experimental setting. Note that there are three CPR applications. The first one is Resusci Anne Simulator [112] manikin on the floor. This is connected to Laderal PC Skillreporting System [111] (version 2.4.1). The second one is the iCPR [40] application (version 1.1), which was installed on an iPhone4S with iOS version 6.1 and was fixed onto the performer arm with an armband. The last one is LISSA, with Kinect (model 1414) connection, set in front of the manikin. Our experiment consists on applying the CPR procedures on the manikin during a minute. All three applications started and stopped at the same time manually. We repeated the experiment five times collecting chest compression performance parameters. Since the Kinect sensor uses the user body as a controller by tracking how the body moves, a calibration step is needed before starting each trial. Calibration consists of holding the hands up and wait until Kinect responds back (as a green bullet 112 Chapter 6. Kinect-based system for LISSA

Figure 6.8: Experiment of CPR-chest compression on manikin and obtaining the chest compression data from LISSA, iCPR and Laderal PC Skillreporting System

Application CPR compression rate Arm position (times per min.) tracking Trial 1 Trial 2 Trial 3 Trial 4 Trial 5 Manual count 122 118 115 115 116 By expert LISSA with Kinect 122 117 95* 115 116 Yes iCPR 122 118 115 115 115 No Laderal PC Skillreporting System 122 118 115 115 116 No

Table 6.3: Experiment result of CPR chest compression rate (times per minute) and correct arm position tracking capability for each application. More details are explained in Section 6.6) 6.6. Discussion and limitations 113 on screen) then positioning the hands onto manikin chest with correct arm posture. At this moment the application is ready to start. To count one minute we used a digital time counter. The obtained results are presented in Table 6.3, where the ﬁrst row shows the number of compressions done by the practitioner in each trial and counted by the expert and the other rows are the number of compressions detected by the different applications. Results show that, except for a large deviation case explained in next section, our system is able to track the compression rate. Remember, as shown in last column of Table 6.3, that it is the only one of the tested criteria that can also check the correct arm position.

6.6 Discussion and limitations

Our system is based on skeleton tracking, which has the following limitations:

Calibration time: In order to detect users and their skeleton, Kinect requires a • few seconds calibration. However, we believe that this is acceptable in practice.

Assumptions on the user pose and position with respect to the Kinect: Hand tracking • (used by both compression counting and arm position) may become less accurate and robust when the hand is in front of the chest or close to other joints, both of which hold during the CPR. In the worst case, skeleton tracking may completely fail by identifying incorrect parts of the body as joints. In our experience, when the arms are carefully and slowly placed on the manikin this occurs only seldom. However, if the calibration process of the correct pose is performed using fast and large arm movements, the system might fail.

Occlusion: Partial occlusion of the user body can greatly degrade skeleton track- • ing performance. In our case, the manikin occludes the user. In our experience, proper alignment of the Kinect can avoid this performance degradation by letting as small part of the manikin into the ﬁeld of view of the device as possible, while still keeping the hand visible.

Sensitivity to the environment: Certain environment conditions such as strong • natural light, movement in both the background or the foreground of the Kinect camera image can have negative effect on the performance of skeleton tracking. As an example, the third experiment in Table 6.3 was disturbed by background movement, resulted in missing a signiﬁcant amount of compressions made by the user.

The process of tracking correct arm position can be provided by marker-based tracking, marker-free (such as Kinect) or haptic devices. Among them, our approach (with Kinect) is the most economical option. We note that the Kinect provides both the raw depth and color images, allowing to develop computer vision algorithms in order to increase the accuracy and robustness of tracking and to enable the evaluation of the rest of the criteria of CPR compression, namely compression depth, correct hand position and placement. We plan to focus, in the future, on computer vision methods, 114 Chapter 6. Kinect-based system for LISSA such as image processing, that requiring as few calibration steps as possible allowing an everyday educational use.

6.7 Conclusion

In this chapter, we have analysed how to improve the physical realism of a serious game. We have considered LISSA as a case study. We have determined the key parameters of the CPR procedure, focusing on the chess compression process, and how they can be solved by a human tracking system. After studying, different alternatives we determined that Kinect is the better one since it provides information in real-time, packed in a compact and lower-cost comparing with other tracking systems. Next, we presented an integration work-ﬂow of Kinect to LISSA focusing on compression rate and correct arm position detection. We have integrated and tested our system and compared it with two other CPR feedback systems: iCPR, a handheld application designed for CPR training coming with a compression counter and Laderal PC Skillreporting System, a PC based CPR learning application connected with Resusci Anne Manikin. Our results show that LISSA with our Kinect based interactive tool is usable regarding to the chest compression rate and the ability to track the correct arm position, albeit with some limitations. As future work, we will introduce several improvements to our system, such as tracking of CPR compression depth, hand placement and patient chest position, to become fully usable. Also further experiments related to the performance of Kinect such as the distance between Kinect and the user or the effect of lighting will be investigated. CHAPTER 7 Evaluation of LISSA as CPR teaching/learning environment

7.1 Introduction

The increasing use of technology over the past decades has changed the way teachers and learners approach their roles in the classroom. In this context, serious game- based learning has emerged as a new approach that satisﬁes the basic requirements of professional skills learning and provides engaging learning experiences for students [124]. In the health area, the application of this new teaching/learning approach, where lessons are learnt by playing, is endorsed by many researchers [32, 52, 170]. In this thesis we have proposed LISSA, a serious game for teaching and learning cardiopulmonary resuscitation skills. Our game has been designed to support the main functionalities of both teachers and learners. For the teacher, LISSA provides functionalities to create new CPR problems modifying a variety of parameters, such as places, avatars, patient information, equipments or to follow-up the student progress by checking the responses to the problems, amongst others. For the learner, it provides a CPR emergency situation that requires to be solved with the CPR protocol. LISSA uses the game challenge and the variety of CPR scenarios, created by instructors, to motivate the game replay in order to acquire CPR skills and knowledge. Focusing on cardiopulmonary resuscitation (CPR), in the last decade, computer simulation has been increasingly applied for teaching/learning this procedure. Different learning strategies have been proposed including learning with interactive video, 3D simulation and learning by playing with serious games. [43, 96, 110, 167, 195, 225] In this chapter we aim at evaluating LISSA from a teaching and learning point of view. We will consider the main users of the application, the instructors and the learners. For the instructors, we will evaluate their perception with regard to teaching context such as usability and teaching functionality. For the learners, we will evaluate their perception with regard to learning context such as ease of learning, satisfaction and learning functionality. In addition, we will compare the learning outcomes obtained with LISSA with respect to manikin based strategies. Our study will be carried out in two different Universities, University of Girona in Spain and Prince of Songkla University in Thailand. 116 Chapter 7. Evaluation of LISSA as CPR teaching/learning environment

7.2 Background

7.2.1 Instruction components

Effective teaching/learning involves three major components, learning objectives, instructional activities, and assessment (see Figure 7.1) [64]. Next, we describe all these components.

Figure 7.1: Major components of instruction [64]

Learning is the process of acquiring or reinforcing knowledge, behaviors, skills, • values, or preferences in different types of information. The learning objectives represent the knowledge and skills that we expect students could demonstrate at the end of the course. There are many different ways to categorize the learning objectives. M. Gagne [77] introduced three learning domains: cognitive, related with mental skills or knowledge; affective related to growth in feelings or emotional areas; and psychomotor, manual or physical skills. B. Bloom [21] proposed the Bloom0s taxonomy which is the most widely applied. Bloom0s taxonomy is composed of six majors levels (remembering, understanding, applying, analysing, evaluating and creating) starting from the simplest behavior to the most complex one (see Figure 7.2). Practically, the ﬁrst level of the taxonomy must be mastered before the next ones can take place. Description and examples of each levels are given in Table 7.1.

Instructional activities are the activities that support learning objectives and teach- • ing. It is a multifaceted activity, aimed at passing the knowledge or experience (skills) to learners. Two types of instruction activities can be considered, direct and indirect. Direct instruction is a general term for the explicit teaching of a skill- set using lectures or demonstrations of the material [236]. This method is often contrasted with tutorials, participatory laboratory classes, discussions, recita- tions, seminars, workshops, observations, case studies, active learning, practice or intern-ships. Indirect instruction is an approach to teach and learn in which concepts, patterns, and abstractions are taught in the context of strategies that 7.2. Background 117

Figure 7.2: Bloom0s taxonomy

Objective Description Example Remembering Recall previous learned informa- defines, describes, identifies, knows, tion labels, lists, matches, names, outlines, recalls, recognizes, reproduces, selects, states Understanding Comprehending the meaning, comprehends, converts, defends, dis- translation, interpolation, and tinguishes, estimates, explains, ex- interpretation of instructions and tends, generalizes, gives an example, problems infers, interprets, paraphrases, pre- dicts, rewrites, summarizes, translates Applying Use a concept in a new situation applies, changes, computes, con- or unprompted use of an abstrac- structs, demonstrates, discovers, mation nipulates, modifies, operates, pre- dicts, prepares, produces, relates, shows, solves, uses Analysing Separates material or concepts analyses, breaks down, compares, into component parts contrasts, diagrams, de-constructs, differentiates, discriminates, distin- guishes, identifies, illustrates, infers, outlines, relates, selects, separates Evaluating Make judgements about the value appraises, compares, concludes, con- of ideas or materials trasts, criticizes, critiques, defends, describes, discriminates, evaluates, explains, interprets, justifies, relates, summarizes, supports Creating Builds a structure or pattern from categorizes, combines, compiles, com- diverse elements poses, creates, devises, designs, explains, generates, modifies, organizes, plans, rearranges, reconstructs, relates, reorganizes, revises, rewrites, summarizes, tells, writes

Table 7.1: Bloom0s taxonomy, learning objective categorization [21] 118 Chapter 7. Evaluation of LISSA as CPR teaching/learning environment

emphasize concept learning, inquiry, and problem solving [55]. In indirect instruction, the learner acquires information by transforming stimulus material into a response that requires the learner to rearrange and elaborate on the stimulus material, such as, self-evaluation, group discussion, use of examples and non-examples, etc. Both direct and indirect instructions can be used in serious games for education or training. The use will be considered during game design procedure and effect directly to the game-play.

Assessment is a process of collecting information and analyse them in order to • get the objective related information. In the education context, assessment can measure student learning and development outcomes, conduction to improve educational programs or a way to demonstrate program effectiveness [75]. There are many styles of assessment (rating) including students rate, peer rate, instructor discusses, self-rates or committee rate.

Focusing on computer aided instruction assessment, in particular on serious games, assessment concept has been proposed by many researchers. Olsen, T. et al. [205] proposed an approach for measuring effectiveness for learning as a col- lective measure of usability, playability and learning outcomes. Ford, and Salas [103] stated that learning can be measured by changes in affective, cognitive, or skill capabilities. In conclusion, the proof of teaching/learning is the main assessment objective. Learning assessment should provide feedback to learners (and instructors) on their progress towards the achievement of learning outcomes. Teaching assessment should provide feedback to instructors (and developers) on the usability (heuristics evaluation) of the application in order to further improve teaching effectiveness. Beside, learner achievement or their outcomes can be one of the parameters presenting teaching effectiveness as well.

The most used usability evaluation for user interface design is the Nielsen0s heuristics [150]. He proposed ten interactive design principles: (i) visibility of system status, keep users informed about what is going on; (ii) match between system and the real world, inform what familiar to the user (e.g. language); (iii) user control and freedom, support undo and redo; (iv) consistency and standards, maintain the same conventions; (v) error prevention, ask before user commit to the action; (vi) recognition rather than recall, minimize what user need to memory; (vii) ﬂexibility and efﬁciency of use, allow users to tailor frequent actions; (viii) aesthetic and minimalist design, remove irrelevant; (ix) help users recognize, diagnose, and recover from errors; and (x) help and documentation,

provide necessary information focused on the user0s task, list of concrete steps to be carried out.

The most common method to assess educational settings are questionnaires [98, 101, 232]. There are also other measurements such as time-on-task or atten- dance rates [207], analysis of facial expressions and body language [157, 176] and observations [50]. 7.2. Background 119

In the context of LISSA, the instruction components can be described as follows. The learning objective is learn and acquire CPR skills to know when and how to apply the different procedures of the CPR protocol. The instructional activities are presented as CPR emergency situations where students have to apply the CPR protocol in a game mode. Our framework covers direct and indirect learning, since it provides information and tutorials of how to apply the protocol and also situations where the student has to apply this protocol. Assessment is obtained from the evaluation of the user actions in the context of the CPR scene. Each action is assessed and compared with the correct protocol. The learner is informed on-line if performed actions are correct or not. When the game ends a performance report is showed, the learner can see the record of his/her decisions during the CPR scenario as well as the performance score provided by LISSA. The instructors can also evaluate learner skills reviewing performance reports.

7.2.2 Manikin based CPR training

Being CPR certiﬁed means having the ability to apply the CPR procedure in the correct way (i.e. proper compression rate, compression depth, ventilation duration and ventilation volume). Certiﬁed CPR courses are available with regard to CPR algorithms provided by many organizations such as Red Cross, American Heart Association (AHA), European Resuscitation Council (ERC) and UK Resuscitation Council. Effec- tive CPR training acquires both theoretically and practically teaching/learning. CPR is commonly taught in classrooms where an expert introduces the main CPR procedures and then practice is done with the support of manikins. This strategy requires the supervision of an expert who controls that the procedures are correctly applied. In addition, some special equipments have been proposed to support CPR teaching. Among

Figure 7.3: (a) Electronic CPR Monitor [83] and (b) Resusci Anne Simulator, a full- body CPR training manikin [112] them, the most recent systems are the Electronic CPR Monitor (see Figure 7.3(a) [83]), which can be connected to a manikin in order to register information such as hand placement, compression depth and ventilation volume, and Resusci Anne Simulator (see Figure 7.3(b) [112], a full-body CPR training manikin with built-in sensors and skill-meter screen that provides comprehensive instant visual feedback. Despite the advantages of these devices, their prices in many situations are prohibitive ($3,000 - $8,000 US). To overcome this limitation, computer-based simulation systems able to 120 Chapter 7. Evaluation of LISSA as CPR teaching/learning environment track CPR performance have become a good alternative.

7.3 LISSA as teaching/learning environment

LISSA exploits video game technology to link in a single framework computer based simulations of CPR emergencies with the functionalities of e-learning platforms. LISSA as a CPR learning tool has applied serious games in order to motivate the repeat learning. Furthermore, LISSA adaptive teaching/learning system can provide various scenarios or emergency problems which instructors can create by themselves (see Chapter 4). LISSA has two main user interfaces, instructors and learners. Within instructor interface, three main pages are available, that are exercise, tutorial and student page. Similar to the instructor interface, learner interface also has three main page, home, tutorial and practice The functionalities of each user interfaces are presented in Table 7.2. In the game mode, learner has to apply the CPR procedure to help a patient in

User Interface Objective Main Functionality Instructor Exercise Exercise or problems - Create new problems management - Edit existing problems - Assign problems to learners Tutorial Tutorial management - Add new tutorials - Edit existing tutorials Student Consult with learners - Review each learners performances (problem solving) - Give comments to learners Learner Home Previous performance - View the list of past performances and the new assignments - View instructor comments Tutorial Theoretical study - View tutorial materials Practice Practical study - View the list of problems in practice mode Exercise Practical study - View the list of problems in exercise mode

Table 7.2: Main functionalities of LISSA the emergency situation. There are three main functions that the learner can apply which are i) perform speciﬁc actions such as open airway, place the patient in recovery position and do chest compression, ii) apply use of medical machines such as automated external deﬁbrillator (AED) for the outside of hospital problem and iii) provide some medications for the advance life support in hospital. For example, Table 7.3 shows available functions for the outside of hospital scenario which allow basic actions and equipment. To use LISSA in the CPR class based training, the instructor has to create problems by selecting the scene elements, such as place, patient character and number of attempt that learners will be allowed to solve, and assign them to the learners. As well as ex- 7.4. Evaluation 121

Place Function Mode Action / Equipment / Medicine Out of hospital Action Patient position Supine, Lateral Response Ask question, Shout, Shake Respiration Open airway, Check breath, Give Breath Execution Remove cloth, Chest compression Misc Ask for help, Call emergency Equipment - AED Medicine - -

Table 7.3: Functions for the outside of hospital problems position materials, of any compatible formats to internet browser, which the instructor would like to introduce to the class also can be uploaded. These problems and exposition materials will be automatically presented to the learners workbooks and once the learners have ﬁnished solving a problem the score will be automatically reported to the instructor in real-time. In this way, LISSA provides an adaptive independent learning that can decrease the training time and can be repeated over time.

7.4 Evaluation

To evaluate LISSA as a teaching/learning environment we considered two different faculties. The ﬁrst one is the Faculty of Inﬁrmary from the University of Girona (UdG). The second one is the Faculty of technology and environment of Prince of Songkla University (PSU) in Thailand. To carry out the experiment we selected two groups of participants, instructors and learners, since we want to evaluate both the teaching and the learning contexts. To evaluate the teaching performance, four CPR instructors (three from UdG and one from PSU) have been invited. To evaluate the learning performance, sixty volunteers were undertaken from PSU with no CPR background. The experiment was carried out between July to August of 2013. In our experiment we considered also the manikin based approach since it is the most common used in class based faculty study, compare with LISSA. In the experiment, LISSA has been set up on personal computers, windows based platform with simply mouse and keyboard interaction.

7.4.1 Teaching evaluation

We use a questionnaire in order to assess LISSA teaching usability. Our questionnaire has three main parts, in the ﬁrst part, general information including gender, age and expertise with media and interfaces on PC video games, were asked. The second part aimed at assessing teaching usability of LISSA. Participants were requested to rate how much they agree, by Likert scale [220], from: strongly disagree, disagree, neutral, agree and strongly agree, about the teaching context of LISSA which are functionality, mobility, data management, ease of teaching, correctness, realism, ease of use and overall satisfaction respectively. With respect to teaching usability, eight questions 122 Chapter 7. Evaluation of LISSA as CPR teaching/learning environment about functionality (Q1), mobility (Q2), management (Q3), ease of learning (Q4), correctness (Q5), realism (Q6), ease of use (Q7) and satisfaction (Q8) have been asked. The questions regard to this test are present in Table 7.4. Finally, in the third part, in order to further improve our application, more depth response in difference perception according to instructor experience is required, thus the instructors were open to comment and advice LISSA in the term of CPR teaching tool.

7.4.2 Learning evaluation

The learning assessment is focused on two contexts, learning outcomes and usability testing. Figure 7.4 illustrates the ﬂow diagram of experiment procedure for learner.

Figure 7.4: Flow diagram of experiment procedure for learner

As we can see in the diagram, first the learning outcomes testing is assessed by comparison score of a randomised cross-over examination between LISSA and manikin- based learning. Sixty of non-CPR certified people were selected in order to maintain the participants0s CPR knowledge at the same level. To start, all participants were requested to do a pre-examination (A) which has two sections. The first section aimed 7.4. Evaluation 123

Exam

Part A: Please ﬁll the correct adult CPR actions (A -H) to the ﬂow diagram in according to the sequence provided by ERC 2010. (A) Ask for help NO NO (B) Observation (C) Emergency call YES YES (D) CPR (E) Check breathing (F) Check responsive (G) Place in recovery position (H) Open airway

Part B: Please choose the correct answer to all following questions.

1. What is the proper way to determine unresponsiveness? A. Pinch their earlobe B. Pour cold water on hand C. Tap and shout, Are you OK? 2. What is the dial number for emergency medical services (EMS) in Europe. A. 911 B. 112 C. 999 3. What is the preferred way to check for breathing? A. Put your ear close to the victim's nose and mouth, and listen for slight breathing. B. Place your hand on the chest and see if it moves with respiration C. Look at the nose and see the sign of breathing 4. What is the best way to open to airway prior to giving mouth to mouth ventilations? A. Tilt the head forward and push down on the neck B. Tile the head back and lift the chin up C. Tile the head back and push down on the chest 5. What is the recovery position? A. Place the victim on his or her side B. Raising the feet up above the heart C. Placing the victims in a sitting position 6. What is the ratio of chest compressions to ventilation in one person adult CPR? A. 15:2 B. 30:1 C. 30:2 7. You have ﬁnished the both chest compression and ventilation already but the patient still has no breath and emergency personnel are not yet arrive. What should you do next? A. Repeating CPR and recheck breathing until help arrive B. Turn the patient to recovery position and recall EMS C. Try to sent the patient to the hospital yourself

Figure 7.5: Exam questions used in our experiment 124 Chapter 7. Evaluation of LISSA as CPR teaching/learning environment at testing the CPR procedure knowledge by asking to fill the correct adult CPR actions to the empty flow diagram of the sequence provided by ERC 2010. The second section are multiple choice questions aiming at asking some specific details in CPR procedure. Exam questions we have used in our experiment is presented in Figure 7.5. Next, participants were randomly assigned to two groups of 30 people. The first group (G1) was taught by traditional CPR learning approach where an expert uses a manikin to introduce the concepts. It was an hour class with 15 students per class. When the class finished the participants did the pre-examination exam again (B). In this way we can compare the learning outcome. Then LISSA was introduced to the participants of this first group. They were allowed to experience LISSA for 20 minutes and ended up with an examination (C). The second group (G2) used LISSA for two rounds of 20 minutes. After a round, they did the exam (D and E) which is also the same as pre-exam one. Finally, the Usability testing had been done via questionnaire (F), by all participants (G1+G2). This questionnaire has two main parts. The first part is aimed at gathering the participant’s characteristics including gender (male/female) and age (from <20 to >60 years old). In the second part, participants were asked to rate how much they agree to the sentence we provided by using the Likert scale [220]. The questions regard to this test are presented in Table 7.6. The possible answers range from: strongly disagree, disagree, neutral, agree and strongly agree.

7.5 Result

The results of the experiment are presented in this section. First we present the teaching and then the learning performance.

7.5.1 Teaching performance

The results of the teaching study come from 4 experts on CPR teaching (three female and one male) who are between 21 to 60 years old. Most of them (75%) use computer more than 10 hours per week and half of them play game less than one hour per week. With respect to teaching usability, eight questions about functionality (Q1), mobility (Q2), management (Q3), ease of learning (Q4), correctness (Q5), realism (Q6), ease of use (Q7) and satisfaction (Q8) have been asked. The results of this experiment are presented in Table 7.4. Figure 7.6 shows evaluation results of all characteristics with regard to the questionnaires as detailed previously. There are three different color lines presented on the spider web graph which are average results (in red) and error polylines (green and blue) of standard deviation. The results show that LISSA has positive performances in CPR teaching context. A half of the assessment contexts (4 out of 8) have been well rated (more than 80%) including: functionality, 86%; ease of teaching, 90%; correctness, 100%; and satisfaction, 96%. The remaining also gained positive score which are: management and ease of use, 76%; mobility, 70% and the lowermost score is 66% (3.3 out of 5) was about the realism degree. 7.5. Result 125

However, participants also comment that LISSA is suitable as a theoretical CPR teaching but not for a practical level since it does not allow the training of a CPR (compression and ventilation) and therefore, it cannot be valued if the students would make the compressions and ventilations in a correct way.

Instructors, n = 4 Questions (1) µ σ (5) Q1 LISSA has all the teaching functions Completely disagree 4.3 ± 0.5 Completely agree I expect it to have ± Q2 My students have the opportunity to Completely disagree 3.5 0.6 Completely agree practice CPR whenever they want ± Q3 The database provided by LISSA is Completely disagree 3.8 0.5 Completely agree well managed ± Q4 It would be easy for me to teach Completely disagree 4.5 0.6 Completely agree CPR using LISSA ± Q5 All procedures in the game mode Completely disagree 5.0 0.0 Completely agree are correct ± Q6 The degree of visual realism in Completely disagree 3.3 0.5 Completely agree LISSA is high enough ± Q7 Learning this interface was easy Completely disagree 3.8 0.5 Completely agree Q8 It is pleasant to use LISSA Completely disagree 4.8 ± 0.5 Completely agree ± Table 7.4: Teaching assessment of LISSA

Figure 7.6: Average of evaluation results (in red); error polylines (green and blue) at one standard deviation with respect to teaching usability

7.5.2 Learning performance

The results of the learning study come from 60 participants who are between 21 to 60 years old. This experiment aimed at comparing the learning outcome by score which 126 Chapter 7. Evaluation of LISSA as CPR teaching/learning environment are presented in Table 7.5. The left table presents the scores of participants in the first group (G1) including pre-examination score in column (A), examination score after manikin based learning in column (B) and finally, examination score of LISSA based learning in column (C). Note that participants have been passed the manikin training before using LISSA in order to assess the performance of LISSA as a CPR complementary learning tool. Similarly, the right table presents the scores of participants in the second group (G2) including pre-examination score in column (A), examination score after first round LISSA based learning in column (D) and finally, examination score of second round LISSA based learning in column (E). Note that, we encourage the second round learning test in order to assess the learning effectiveness of LISSA as a CPR learning tool that can be repeated using over time.

Figure 7.7: The learning outcomes of differences CPR learning styles (A-E)

The results are summarized and illustrated in Figure 7.7. From this ﬁgure, we can see that the learning result, by compare with the pre-examination score, of manikin (37%) is greater than LISSA (22%) Manikin based study (B) has higher effectiveness than LISSA (D) in the term of learning outcome which is assessed by examination score (B=67.67% and D=52.89%),14.78%. Although LISSA had a lower outcome than manikin based learning, it can be used as a complementary CPR learning tool since the examination score of participants who used manikin followed by LISSA (E) is greater than the ones that only used manikin. Furthermore, LISSA has a potential to improve the learning outcomes according to the experiment result state that the examination score of 2nd round using LISSA (E) is greater than the 1st round. The difference is around 6.67%. Figure 7.8 illustrate the improvement of learning outcome for each participant. The top chart presents the improvement by learning with manikin (B) and followed by LISSA (C) while the bottom chart presents the improvement by learning with LISSA in the ﬁrst round (D) and 2nd round (E) from the score that participants obtained before learning (A). The results of the learning usability experiment are presented in Table 7.6. The 7.5. Result 127

Full mark =15 Group1 Pre-exam Manikin LISSA Group2 Pre-exam LISSA LISSA (G1) (A) (B) (C) (G2) (A) (D) (E) P1 1 7 10 P1 2 5 7 P2 2 13 13 P2 2 6 6 P3 3 9 12 P3 3 4 6 P4 3 11 12 P4 3 7 9 P5 3 9 10 P5 3 9 11 P6 3 7 9 P6 3 10 9 P7 3 10 10 P7 4 5 6 P8 3 10 11 P8 4 5 6 P9 4 8 11 P9 4 6 10 P10 4 9 10 P10 4 7 9 P11 4 8 11 P11 4 7 11 P12 4 11 10 P12 4 8 10 P13 4 10 12 P13 4 10 10 P14 4 10 13 P14 4 10 11 P15 5 15 15 P15 5 6 7 P16 5 8 9 P16 5 6 7 P17 5 8 10 P17 5 7 8 P18 5 9 10 P18 5 8 8 P19 6 13 15 P19 5 9 8 P20 6 8 10 P20 5 10 10 P21 6 11 12 P21 5 10 10 P22 6 11 11 P22 5 10 9 P23 6 9 11 P23 6 6 9 P24 6 10 10 P24 6 7 9 P25 6 10 10 P25 6 9 9 P26 6 11 10 P26 6 10 13 P27 6 13 13 P27 6 11 10 P28 6 12 12 P28 6 12 12 P29 6 9 10 P29 7 8 8 P30 7 12 13 P30 8 10 10 (µ) 4.60 10.0 11.17 (µ) 4.63 7.93 8.93 (%) 30.67% 67.67% 74.47% (%) 30.89% 52.89% 59.56%

Table 7.5: Examination score (where A is pre-examination score, B is post-examination of mannequin, C is post-examination of LISSA after B, D is post-examination of LISSA and E is post-examination of mannequin after D) 128 Chapter 7. Evaluation of LISSA as CPR teaching/learning environment

Figure 7.8: (Top) The improvement of learning outcome for the participant in G1 and (Bottom) The improvement of learning outcome for the participant in G2 7.6. Conclusion 129 answers to the ten questions about joy (Q1), ease of use (Q2), comfortability (Q3), functionality (Q4), effectiveness (Q5), usefulness (Q6), ease of learning (Q7), satisfaction (Q8), timing (Q9) and mobility (Q10) are unanimous positive. Figure 7.9 shows evaluation results of all characteristics with regard to the questionnaires as detailed previously. There are three different color lines presented on the spider web graph which are average results (in red) and error polylines (green and blue) of standard deviation. From the ﬁgure, we can see that LISSA has positive performances in CPR learning context. Three out of ten assessment contexts have been well rated (more than 80%) which are: effectiveness, 84%; usefulness, 82% and mobility, 82%. The remaining also gained positive score which are: joy, 76%; functionality, satisfaction and timing, 74%; and ﬁnally, ease of use, comfortability and ease of learning, 72%.

Learners, n = 60 Questions (1) µ σ (5) Q1 Using this interface was enjoyable Completely disagree 3.8 ± 0.7 Completely agree Q2 Learning this interface was easy Completely disagree 3.6 ± 0.6 Completely agree Q3 This interface was comfortable to use Completely disagree 3.6 ± 0.7 Completely agree Q4 My intended actions were accurately Completely disagree 3.7 ± 0.7 Completely agree carried out on screen when using ± this interface Q5 I feel that more practice time with this Completely disagree 4.2 0.6 Completely agree interface would have made a signiﬁcant ± difference to my performance Q6 My CPR-knowledge can be improved Completely disagree 4.1 0.6 Completely agree by using LISSA ± Q7 I quickly became skillful with LISSA Completely disagree 3.6 0.8 Completely agree Q8 It is pleasant to use LISSA Completely disagree 3.7 ± 0.7 Completely agree Q9 I am satisﬁed with the amount of time Completely disagree 3.7 ± 0.6 Completely agree it took to complete the tasks in this ± scenario Q10 LISSA is a mobility CPR learning tool Completely disagree 4.1 0.4 Completely agree which I can use anywhere and ± anytime

Table 7.6: Learning assessment of LISSA

7.6 Conclusion

In this chapter, we have assessed our CPR application, LISSA, on its teaching/learning performance. Our experiments have been carried out at the inﬁrmary faculty of Uni- versity of Girona in Spain and the faculty of technology and environment of Prince of Songkla University in Thailand. The teaching assessment has been done with four CPR instructors. The results show the positive perception on eight teaching contexts including functionality, mobility, management, ease of learning, correctness, realism, ease of use and satisfaction. The version of LISSA for this experiment is based on mouse and keyboard, thus the participants comment that since LISSA does not allow learner to do a physical action for CPR main maneuver (compression and ventilation) then it is 130 Chapter 7. Evaluation of LISSA as CPR teaching/learning environment

Figure 7.9: Average of evaluation results (in red); error polylines (green and blue) at one standard deviation with respect to learning usability more suitable to consider LISSA as a theoretical CPR teaching. To solve this problem, we have proposed the use of Kinect as an interaction tool to improve the physical realism of LISSA (more details are presented in Chapter 6). The learning assessment has been done with sixty participants. The results shows that LISSA is working well as a CPR complementary learning tool since the learning outcome can be increased when participants use LISSA after manikin based class. Beside, LISSA shows an improvement of learning outcome when reuse it. Focusing on learning perception, the results show the positive perception on ten learning contexts including joy, ease of use, comfortability, functionality, effectiveness, usefulness, ease of learning, satisfaction, timing and mobility. The use of computer-based simulation following mannequin-based simulation can improved the achievement of teaching/learning goals and outcomes. In addition, to maintain CPR knowledge and skills with completing the computer-based simulation is and effective way and may improve the conﬁdence during the real life CPR performance in the emergency situation. CHAPTER 8 Conclusion

Serious games, video games designed for speciﬁc objectives, have a clear potential to support the training and education of health and medical for the large public. The study of serious games for health and medical has become an emerging focus of research since it can provide the interaction, simulation and required information along with the advancement of technology. In this thesis, we have focused our interest on exploiting serious game technology to enhance teaching/e-learning in health environments, speciﬁcally on cardiopulmonary resuscitation (CPR). Below, the detailed description of the main contributions of the thesis as well as the publications related to each contribution are given.

8.1 Contribution and publication

The main goal of this thesis has been the study of serious games for health. To approach our study, we have developed a serious game, Life support simulation application (LISSA), focused on cardiopulmonary resuscitation learning. Our contributions can be summarized as follows:

We have surveyed more than one hundred serious games for health and proposed • new classifications in four different aspects and their functionalities. We have used fifteen relevant characteristics to classify all the surveyed games. This work has lead to two publications. The first paper titled Serious games for e-healthcare that has been published as a chapter in a Springer Book of Asian- European Workshop on Serious Game and Simulation, 25th Annual Conference on Computer Animation and Social Agents (CASA 2012), Singapore, 9-11 May 2012. The second contribution, an extension of the first one and titled Serious games for health, has been accepted to the Entertainment Computing Journal, Elsevier, Ref. No.: ENTCOM-D-13-00008R1, Sep 2013. This extension includes an explanation of serious games core components and 80 more surveyed serious games with discussions.

We have developed a serious game to teach and learn CPR skills. This game, de- • noted LISSA (LIfe Support Simulation Application) combines in a single framework computer based simulations of CPR emergencies with the functionalities of e-learning platforms. Different emergency scenarios can be created by the instructor and presented as problems that the learner has to solve in a game 132 Chapter 8. Conclusion

mode. All the actions and all the cases are registered in a database that can be consulted at any moment by the instructor. LISSA can be used as a substitute or a complement for traditional CPR classroom-based instruction. The adaptive learning supported by LISSA makes it suitable to refresh and improve CPR skill retention over time. This work has lead to two different publications. The ﬁrst paper titled The framework of a LIfe Support Simulation Application, has been published in the proceedings of 4th International Conference on Games and Virtual Worlds for Serious Applications (VS GAMES12), 29 Oct to 1Nov, 2012. Genoa, Italy and the journal of Procedia Computer Science, Volume 15, 2012, Pages 293 - 294, ISSN 1877- 0509 The second contribution, an extension of the ﬁrst one with the work titled LISSA: A Serious Game to learn Cardiopulmonary Resuscitation, that has been published in the proceedings of the 8th international conference on Foundations of Digital Games (FDG2013): Games for Learning workshop, 14 -17 May 2013, Chania, Crete, Greece. This extension includes more explanations with regard to additional functionalities such as the score system.

We have studied the elements and process of visual realism design for 3D serious • games, considering LISSA as a case study. Realism is one of the main factors which motivates players to accomplish their goals in a serious game. An evaluation of visual realism with regard to photorealism and camera viewpoint has been presented. This work has lead to a publication titled Visual realism in 3D serious games for learning: A case study, that has been published in the proceedings of 15th Interna- tional Conference on Enterprise Information Systems: Workshop on Interaction Design in Educational Environments (ICEIS 2013), July 5, 2013, Angers, France.

We have presented a pilot study in developing a Kinect-based system for our ex- • isting LIfe Support Simulation application (LISSA). Focusing on two key parameters of the CPR procedure (the chest compression rate and correct arm position), we have evaluated different CPR feedback systems and described how Kinect has been integrated in LISSA. We also presented the experiments that have been done to evaluate it. We further compared three CPR feedback systems with regard to the chest compression rate and correct arm position. The results appear in the paper titled A Kinect-based System for Cardiopulmonary Resuscitation Simulation: A pilot study, which will be published in the proceedings of 4th International Conference on Serious Games Developments and Appli- cations (SGDA 2013), 25 - 27 Sep, 2013, Trondheim, Norway and will appear in the Springer Lecture Notes in Computer Science (LNCS) series LNCS 8101.

We have evaluated LISSA in a real teaching/learning context and considered • the two main target players, CPR instructors and learners. For the learners, the knowledge improvement has been tested (by pre and post-examination score), 8.2. Future work 133

in addition to an assessment on learn ability and player perception, compared to learning with manikin (under control of the experts). For the instructors, the group interview of instructors from Inﬁrmary Faculty at University of Girona, has been conducted. Positive results have been obtained from both instructors and learners. This work has been collected in the paper titled LISSA, a new e-learning environment to learn CPR, which has been submitted to the Journal of Resuscitation, 2013

8.2 Future work

Our future works will be focused on the following research lines:

Extend the LISSA environment with new characters and scenes as well as the • functionality of adaptive learning system, to deal with any kind of life support procedure such as in-hospital resuscitation, neonatal and paediatric life support including advanced life support with several algorithms that require medication.

Investigate on functional realism as well as the further possibilities of visual re- • alism that effects each step of learning process.

Improve the physical realism of our system, such as tracking of CPR compression • depth, hand placement and patient chest position, to become fully usable by considering different human motion tracking devices including optical and non- optical based systems. Also further experiments related to the performance of Kinect such as the distance between Kinect and the user or the effect of lighting will be investigated.

Extend the study with more learners and instructors, we plan to use LISSA as a • basic element in a CPR course of Inﬁrmary Faculty at University of Girona.

Bibliography

[1] Virtual reality and mixed reality for virtual learning environments. Computers & Graphics, 30(1):20 – 28, 2006. (Cited on page 76.)

[2] Evaluation of m-aidÂ R , a ﬁrst aid application for mobile phones. Resuscitation, 74(3):487 – 494, 2007. (Cited on page 66.)

[3] Product interface design: A participatory approach based on virtual reality. In- ternational Journal of Human-Computer Studies, 68(5):254 – 269, 2010. (Cited on page 76.)

[4] Agusta Westland. Helismart (vmt). "http://www.agustawestland.com/ content/virtual-maintenance-trainer-vmt", 2012. Accessed 26 Jul 2013. (Cited on page 7.)

[5] E. Alhadeff. Pulse!! news: Serious games ﬁeld testing begins. http://futuremakingseriousgames.blogspot.com/2007/02/pulse- news-serious-games-field-testing.html, 2007. Accessed 7 March 2012. (Cited on pages 32 and 47.)

[6] American Heart Association. Emergency Cardiovascular Care online education. "http://www.onlineaha.org", 2012. Accessed 30 May 2012. (Cited on pages 65 and 66.)

[7] Anchor Bay Entertainment. 10 Minute Solution for Wii. "http://www. the10minutesolution.com/wii/index.html", 2010. Accessed 23 Novem- ber 2012. (Cited on page 36.)

[8] C. Anderson. Simulation game playing: A nursing instructional strategy. Clinical Simulation in Nursing, 4(1):e7 – e15, 2008. (Cited on page 33.)

[9] Anne Derryberry, Adobe. Serious games: online games for learning. "http://www.adobe.com/resources/elearning/pdfs/serious_ games_wp.pdf", 2007. Accessed 29 Jul 2013. (Cited on pages 7 and 16.)

[10] Applied Research Associates, Inc. HumanSim: a high-ﬁdelity virtual hospital. "http://www.humansim.com", 2012. Accessed 6 March 2012. (Cited on pages 40 and 47.)

[11] Archimage, Inc. Escape from Diab. "http://www.escapefromdiab.com/", 2006. Accessed 4 January 2013. (Cited on page 31.)

[12] Association RMC, BFM. Staying alive. "http://www.stayingalive.fr", 2011. Accessed 19 November 2012. (Cited on page 38.) 136 Bibliography

[13] S. Atkinson and V. Narasimhan. Design of an introductory medical gaming environment for diagnosis and management of parkinson disease. In Trendz in Information Sciences and Computing (TISC), pages 94–102, May 2010. (Cited on pages 36 and 46.)

[14] B2B Games and Persistant Studios. 3d networks. "http://www.b2b-games. com/3dreseaux", 2009. Accessed 20 Aug 2013. (Cited on page 18.)

[15] N. Bartolome, A. Zorrilla, and B. Zapirain. Can game-based therapies be trusted? Is game-based education effective? A systematic review of the Serious Games for health and education. 16th International Conference on Computer Games (CGAMES), 3:275–282, July 2011. (Cited on page 29.)

[16] N. Bartolome and, A. Zorrilla, and B. Zapirain. A serious game to improve human relationships in patients with neuro-psychological disorders. Games In- novations Conference (ICE-GIC), pages 1–5, December 2010. (Cited on pages 36, 45 and 46.)

[17] BBG Entertainment GmbH. "Train your Brain" with Dr. Kawashima. "https://itunes.apple.com/gb/app/train-your-brain-dr.- kawashima/id416798601?mt=12", 2009. Accessed 23 November 2012. (Cited on page 34.)

[18] Believe in Tomorrow National Children Foundation, Inc. FREEDIVE. "https: //www.trademarkia.com/freedive-78671194.html", 2006. Accessed 4 January 2013. (Cited on page 31.)

[19] G. Bleser and D. Stricker. Advanced tracking through efﬁcient image processing and visual-inertial sensor fusion. In Virtual Reality Conference, 2008. VR ’08. IEEE, pages 137–144, 2008. (Cited on page 105.)

[20] Blitz Games Studios, Ltd. The Biggest Loser. "http://www.blitzgames. com", 2009. Accessed 23 November 2012. (Cited on page 34.)

[21] B. Bloom. Taxonomy of educational objectives. Handbook I: The Cognitive Do- main. New York: David McKay Co Inc., 1956. (Cited on pages 15, 116 and 117.)

[22] C. Botella, J. Breton-Lopez, S. Quero, R. Banos, A. GarcÃa-Palacios, I. Zaragoza, and M. Alcaniz. Treating cockroach phobia using a serious game on a mobile phone and augmented reality exposure: A single case study. Computers in Hu- man Behavior, 27(1):217–227, 2011. (Cited on page 38.)

[23] C. Bouten, K. Koekkoek, M. Verduin, R. Kodde, and J. Janssen. A triaxial accelerometer and portable data processing unit for the assessment of daily physical activity. Biomedical Engineering, IEEE Transactions on, 44(3):136 –147, march 1997. (Cited on page 108.) Bibliography 137

[24] M. Boyer. First Aid & CPR Basics, First Aid for Choking Victims. "http://www. youtube.com/watch?v=htzRcv1Z6_s&feature=relmfu", 2007. Accessed 29 May 2012. (Cited on page 65.)

[25] BreakAway, Ltd. Serious games for healthcare markets. "http://www. breakawaygames.com/serious-games/solutions/healthcare/", 2010. Accessed 6 March 2012. (Cited on pages 32, 36, 43 and 47.)

[26] Z. Budrikis. Visual ﬁdelity criterion and modeling. Proceedings of the IEEE, 60 (7):771 – 779, july 1972. (Cited on page 23.)

[27] J. W. Burke, M. McNeill, D. Charles, P.Morrow, and S. Crosbie, J. McDonough. Serious games for upper limb rehabilitation following stroke. 2009 Conference in Games and Virtual Worlds for Serious Applications, page 103, 2009. (Cited on pages 34 and 46.)

[28] M. Cagatay, P. Ege, G. Tokdemir, and N. Cagiltay. A serious game for speech disorder children therapy. In 7th International Symposium on Health Informatics and Bioinformatics (HIBIT), pages 18 –23, april 2012. (Cited on pages 40, 45 and 46.)

[29] Caspian Learning. Vehicle recognition training : Virtual task simulator. "http://www.caspianlearning.co.uk/portfolio-item/joint- recognition-training", 2012. Accessed 26 Jul 2013. (Cited on page 7.)

[30] CCCP. LudoMedic. "http://www.ludomedic.com/", 2011. Accessed 19 November 2012. (Cited on page 38.)

[31] N. I. Centre. Energiasummaaja. "http://nsg.jyu.fi/index.php/ Energiasummaaja", 2007. Accessed 26 March 2012. (Cited on page 32.)

[32] W. Chan, J. Qin, Y. Chui, and P.Heng. A serious game for learning ultrasound- guided needle placement skills. IEEE Transactions on Information Technology in Biomedicine, PP(99):1, 2012. (Cited on pages 40 and 115.)

[33] Y.-J. Chang, S.-F. Chen, and J.-D. Huang. A kinect-based system for physical rehabilitation: A pilot study for young adults with motor disabilities. Research in Developmental Disabilities, 32(6):2566 – 2570, 2011. (Cited on page 106.)

[34] X. Cheng-yong and X. Wei-ming. Constructing 3d game engine based on xna. Computer Knowledge and Technology, 6:3401–2, May 2010. (Cited on page 44.)

[35] R. A. Clark, Y.-H. Pua, K. Fortin, C. Ritchie, K. E. Webster, L. Denehy, and A. L. Bryant. Validity of the microsoft kinect for assessment of postural control. Gait & Posture, 36(3):372 – 377, 2012. (Cited on page 106.)

[36] J. Clawson, N. Patel, and T. Starner. Dancing in the streets: The design and evaluation of a wearable health game. International Symposium on Wearable Computers (ISWC), pages 1–4, 2010. (Cited on pages 36 and 47.) 138 Bibliography

[37] College to become an EMT. How to Perform CPR. "http://www.youtube. com/watch?v=cPEFskCrdhQ", 2008. Accessed 29 May 2012. (Cited on page 65.)

[38] Collision Studios. Daisy Fuentes Pilates. "http://www.gamespot.com/ daisy-fuentes-pilates", 2009. Accessed 23 November 2012. (Cited on page 34.)

[39] R. Cross. Interactive realism for visualization using ray tracing. In Visualiza- tion, 1995. Visualization ’95. Proceedings., IEEE Conference on, pages 19–26, 434, 1995. (Cited on page 23.)

[40] D-Sign S.r.l. iCPR - iPhone App for CPR training. "http://www.icpr.it/", 2009. Accessed 19 Dec 2012. (Cited on pages 66 and 111.)

[41] E. Dale. Audiovisual methods in teaching. third edition. New York: The Dryden Press;. Holt, Rinehart and Winston. Dwyer, fm, 1969. (Cited on pages 11, 16 and 17.)

[42] R. L. M. Dariush Derakhshani. Introducing 3ds Max 2008. John Wiley and Sons. p. 240., 2008. (Cited on page 18.)

[43] Dassault SystÃ¨mes - Ilumens - Medusims. Staying Alive 3D. "http://www. stayingalive.fr/index_uk.html", 2011. Accessed 15 Dec 2012. (Cited on pages 69 and 115.)

[44] DDRGame.com. Dance Dance Revolution. "http://www.ddrgame.com/", 2012. Accessed 29 March 2012. (Cited on page 42.)

[45] A. De Bortoli and O. Gaggi. Playwitheyes: A new way to test children eyes. In IEEE 1st International Conference on Serious Games and Applications for Health (SeGAH), pages 1 –4, nov. 2011. (Cited on pages 39, 45 and 46.)

[46] Z. de Urturi, A. Zorrilla, and B. Zapirain. Serious game based on ﬁrst aid education for individuals with autism spectrum disorder (asd) using android mobile devices. In 16th International Conference on Computer Games (CGAMES), pages 223 –227, july 2011. (Cited on pages 40 and 47.)

[47] D. Deponti, D. Maggiorini, and C. Palazzi. Droidglove: An android-based application for wrist rehabilitation. In International Conference on Ultra Mod- ern Telecommunications Workshops (ICUMT), pages 1–7, oct. 2009. (Cited on pages 7 and 34.)

[48] Designing Digitally, Inc. Renault academy; ultimate sales manager. "http: //www.daesign.com/en/realisations/renault-academy.html", 2012. Accessed 26 Jul 2013. (Cited on page 7.) Bibliography 139

[49] Designing Digitally, Inc. Fireﬁghter training simulation, revas process. "http://www.designingdigitally.com/portfolio/simulations/ firefighter-training-simulation-revas-process", 2013. Accessed 26 Jul 2013. (Cited on page7.)

[50] N. Diah, M. Ismail, S. Ahmad, and M. Dahari. Usability testing for educational computer game using observation method. In Information Retrieval Knowledge Management, (CAMP), 2010 International Conference on, pages 157–161, 2010. (Cited on page 118.)

[51] L. Diehl, E. Lehmann, R. Souza, J. Alves, R. Esteves, and P. Gordan. A serious game prototype for education of medical doctors and students on insulin management for treatment of diabetes mellitus. In IEEE 1st International Conference on Serious Games and Applications for Health (SeGAH), pages 1–4, nov. 2011. (Cited on page 39.)

[52] I. Dunwell, P.Petridis, S. Arnab, A. Protopsaltis, M. Hendrix, and S. de Freitas. Blended game-based learning environments: Extending a serious game into a learning content management system. In Third International Conference on In- telligent Networking and Collaborative Systems (INCoS), 2011, pages 830–835, 2011. (Cited on page 115.)

[53] e-Learning Studios. iSpectrum. "http://www.ispectrum.eu/", 2012. Ac- cessed 19 November 2012. (Cited on page 41.)

[54] Edheads. Virtual knee replacement surgery. "http://www.edheads.org/ activities/knee/index.shtml", 2007. Accessed 23 November 2012. (Cited on page 32.)

[55] P. Education. Teaching Strategies for Indirect Instruction. "http://wps. prenhall.com/chet_borich_effective_6/48/12538/3209831.cw/- /3209833/index.html", 2010. Accessed 11 Sep 2013. (Cited on page 118.)

[56] Electronic Arts, Inc. EA SPORTS Active 2. "http://www.ea.com/ea- sports-active-2", 2010. Accessed 23 November 2012. (Cited on pages 37 and 106.)

[57] EM Gladiators LLC. CPR Game. "http://www.freenew.net/iphone/cpr- game-20/466678.htm", 2011. Accessed 15 Dec 2012. (Cited on page 66.)

[58] EMCO3. MD Advisor, The new serious game for medical students. "http://www.aprv.eu/index.php?option=com_content&view= article&id=563:md-advisor-le-nouveau-serious-game-sur- le-theme-de-la-sante-de-emco3&catid=8:actu&Itemid=58", 2012. Accessed 19 November 2012. (Cited on page 41.)

[59] Engineering and Computer Simulations (ECS). Cruise ship evacuation sim. "http://www.ecsorl.com/news/innovative-orlando-based- 140 Bibliography

simulation-company-takes-maiden-voyage-cruise-line-market- interactive", 2012. Accessed 20 Aug 2013. (Cited on page 18.)

[60] Eric Johnston and Ben Duskin. Ben Game. "http://www.makewish.org/ site/pp.asp?c=cvLRKaO4E&b=64401", 2004. Accessed 4 January 2013. (Cited on page 31.)

[61] Estambril Siqur Project sl. Siqur. "http://www.siqur.com", 2011. Accessed 26 Jul 2013. (Cited on page 7.)

[62] Eurogamer expert, Ltd. Games industry jobs. "http://www.gamesindustry. biz/jobs", 2002-2012. Accessed 20 August 2012. (Cited on page 14.)

[63] European Resuscitation Council. Guidelines for Resuscitation 2010. "http: //www.cprguidelines.eu/2010/", 2010. Accessed 7 May 2012. (Cited on pages 13, 66, 72, 74 and 107.)

[64] T. Excellence and E. Innovation. Principles of teaching. "http://www. cmu.edu/teaching/principles/teaching.html", 2013. Accessed 11 Sep 2013. (Cited on pages 14 and 116.)

[65] J. Fabian, T. Young, J. C. P. Jones, and G. M. Clayton. Integrating the microsoft kinect with simulink: Real-time object tracking example. Mechatronics, IEEE/ASME Transactions on, PP(99):1 –9, 2012. (Cited on page 106.)

[66] Faculty of Medicine, Imperial College London. Game-based learning for Virtual Patients - Multi patients. "http://www.youtube.com/watch?v= WnPYhSbSABA", 2008. Accessed 29 January 2012. (Cited on page 33.)

[67] Fatworld.org. Fatworld. "http://fatworld.org/", 2007. Accessed 26 March 2012. (Cited on pages 32, 43 and 47.)

[68] P. Fergus, K. Kifayat, S. Cooper, M. Merabti, and A. El Rhalibi. A framework for physical health improvement using wireless sensor networks and gaming. 3rd International Conference on Pervasive Computing Technologies for Healthcare, pages 1–4, 2009. (Cited on page 46.)

[69] J. A. Ferwerda. Three varieties of realism in computer graphics. In Proceedings SPIE Human Vision and Electronic Imaging, volume 3, pages 290–297, 2003. (Cited on pages 11 and 22.)

[70] J. Finkelstein, J. Wood, E. Cha, A. Orlov, and C. Dennison. Feasibility of con- gestive heart failure telemanagement using a wii-based telecare platform. IEEE Engineering in Medicine and Biology Society, pages 2211–4, 2010. (Cited on page 46.)

[71] S. Finkelstein, A. Nickel, T. Barnes, and E. Suma. Astrojumper: Designing a virtual reality exergame to motivate children with autism to exercise. In Virtual Reality Conference (VR), IEEE, pages 267 –268, march 2010. (Cited on page 37.) Bibliography 141

[72] FirstAidWeb Inc. Learn CPR. For Life. "http://www.firstaidweb.com", 2012. Accessed 30 May 2012. (Cited on page 65.)

[73] J. M. Fisher. ABC of resuscitation. Recognising a cardiac arrest and providing basic life support. British Medical Journal (Clinical Research Ed.), April 12:1002– 4, 1986. (Cited on page 65.)

[74] Fishing Cactus. Fishing Cactus presents R.O.G.E.R, the ﬁrst Medical Kinect Serious game. "http://blog.fishingcactus.com/index.php/2010/ 10/07/fishing-cactus-presents-r-o-g-e-r-the-first-medical- kinect-serious-game/", 2010. Accessed 19 November 2012. (Cited on pages 27, 37 and 106.)

[75] C. for Assessment and J. M. U. Research Studies. The program assessment support service. "http://www.jmu.edu/assessment/pass/ assmntresources/what_is_assessment.htm", 2013. Accessed 12 Sep 2013. (Cited on page 118.)

[76] A. Fushslocher, J. Niesenhaus, and N. Kramer. Serious games for health: An empirical study of the game balance for teenagers with diabetes mellitus. En- tertainment Computing, 2(2):97–101, 2011. (Cited on page 39.)

[77] R. M. Gagne. The conditions of learning. New York: Holt, Rinehart and Winston., 1985. (Cited on page 116.)

[78] J. Gago, T. Barreira, R. Carrascosa, and P. Segovia. Nutritional serious-games platform. In eChallenges, pages 1 –8, oct. 2010. (Cited on pages 37 and 47.)

[79] Gameloft. Brain Challenge. "http://www.gameloft.com/iphone-games/ brain-challenge", 2006. Accessed 4 January 2013. (Cited on page 31.)

[80] C. Gatzidis, K. Parry, E. Kavanagh, A. Wilding, and D. Gibson. Towards the development of an interactive 3d coach training serious game. In Conference in Games and Virtual Worlds for Serious Applications (VS-GAMES)., pages 186 –189, march 2009. (Cited on page 15.)

[81] GENIOUS Interactive. Voracy Fish: New multiplayer serious game for physical rehabilitation. "http://www.voracyfish.fr/", 2012. Accessed 19 Novem- ber 2012. (Cited on pages 41 and 106.)

[82] Glasgow Caledonian University. Nurse education in second life at Glas- gow Caledonian University [demo]. http://www.youtube.com/watch?v= xidko60S2Uk, 2008. Accessed 7 March 2012. (Cited on page 33.)

[83] Global Technologies. CPARLENE Electronic Monitors. "http://www. gtsimulators.com/product-p/lf03401u.htm", 2013. Accessed 6 Feb 2013. (Cited on pages 14 and 119.) 142 Bibliography

[84] L. Gostin. A public health approach to reducing error: medical malpractice as a barrier. JAMA, 283(13):1742–3, 2000. (Cited on page 7.)

[85] S. Grau, D. Tost, R. Campeny, S. Moya, and M. Ruiz. Design of 3d virtual neuropsychological rehabilitation activities. Second International Conference on Games and Virtual Worlds for Serious Applications, pages 109–16, 2010. (Cited on pages 37 and 46.)

[86] D. Hatﬁeld. First Look: Allen Carr Easyway to Stop Smoking. Are you a quitter? "http://www.ign.com/articles/2008/05/28/first-look- allen-carrs-easyway-to-stop-smoking", 2008. Accessed 23 November 2012. (Cited on page 33.)

[87] HCI Lab, University of Udine. The EMSAVE System. "http://hcilab.uniud. it/soccorsodisabili/results.html", 2011. Accessed 7 March 2012. (Cited on pages 38, 43 and 47.)

[88] HopeLab. Zamzee: A game that gets families moving. "https://www.zamzee. com", 2010. Accessed 20 November 2012. (Cited on page 37.)

[89] J. Housman and S. Dorman. The alameda county study: A systematic, chrono- logical review. American Journal of Health Education, 36:302–8, Sept/Oct 2005. (Cited on page 47.)

[90] W. Hu, Q. Zhang, and Y. Mao. Component based hierarchical state machine; a reusable and ﬂexible game ai technology. In Information Technology and Artiﬁ- cial Intelligence Conference (ITAIC), 2011 6th IEEE Joint International, volume 2, pages 319 –324, aug. 2011. (Cited on page 71.)

[91] J. Huizinga. Homo ludens : a study of the play-element in culture. Boston : Beacon Press, 1955, c1950. (Cited on page 11.)

[92] F.Imbeault, B. Bouchard, and A. Bouzouane. Serious games in cognitive training for alzheimer patients. In IEEE 1st International Conference on Serious Games and Applications for Health (SeGAH), pages 1–8, nov. 2011. (Cited on pages 39 and 47.)

[93] Indiegogo Inc. Mini-VREM. "http://www.indiegogo.com/mini-vrem", 2012. Accessed 15 Dec 2012. (Cited on page 69.)

[94] InhalothÃ c rapie KinÃ c sithÃ c rapie respiratory Association for Research and Education (IKARE). MUCOPlay. "http://mucoplay.org/", 2012. Ac- cessed 19 November 2012. (Cited on page 41.)

[95] Innovation in Learning, Inc. CliniSpace. "http://www.clinispace.com/", 2010. Accessed 20 November 2012. (Cited on page 37.)

[96] Insight Instructional Media LLC. AED Challenge. "http://aedchallenge. com/", 2011. Accessed 15 Dec 2012. (Cited on pages 69 and 115.) Bibliography 143

[97] Intelligent System Co., Ltd., Nintendo Software Planning and Development (SPD). Otona no DS Kao Training. "http://www.mobygames.com/game/ otona-no-ds-kao-training-/", 2007. Accessed 26 November 2012. (Cited on page 32.)

[98] M. Ismail, N. Diah, S. Ahmad, N. Kamal, and M. Dahari. Measuring usability of educational computer games based on the user success rate. In Humanities, Sci- ence Engineering Research (SHUSER), 2011 International Symposium on, pages 56–60, 2011. (Cited on page 118.)

[99] V.Janarthanan. Serious video games: Games for education and health. In Ninth International Conference on Information Technology: New Generations (ITNG), pages 875 –878, april 2012. (Cited on page 15.)

[100] Janomedia. Terveellinen ateria. "http://www03.edu.fi/ oppimateriaalit/healthy_meal", 2006. Accessed 26 March 2012. (Cited on page 31.)

[101] K. Jegers. Investigating the applicability of usability and playability heuristics for evaluation of pervasive games. In Internet and Web Applications and Services, 2008. ICIW ’08. Third International Conference on, pages 656–661, 2008. (Cited on page 118.)

[102] P.Jensen. At this end of the oregon trail: The eugene augur: 1969-1974. Serials Review, 16(3):91 – 95, 1990. (Cited on page 11.)

[103] E. S. K. Kraiger, J. K. Ford. Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation, 1993. (Cited on page 118.)

[104] F. Kahlesz, G. Zachmann, and R. Klein. ’visual-ﬁdelity’ dataglove calibration. In Computer Graphics International, 2004. Proceedings, pages 403 –410, june 2004. (Cited on page 23.)

[105] P. M. Kato. Video games in health care: Closing the gap. Review of General Psychology, 14:113–121, 2010. (Cited on page 29.)

[106] J.-A. Kim, K.-K. Kang, H.-R. Yang, and D. Kim. A sensory gate-ball game for the aged people and its user interface design. Conference in Games and Vir- tual Worlds for Serious Applications, pages 111–116, 2009. (Cited on pages 35 and 47.)

[107] G. Kost. Preventing medical errors in point-of-care testing: security, validatis, safeguards, and connectivity.. Arch Pathol Lab Med, 125(10):1307–15, 2001. (Cited on page 7.)

[108] W. B. Kouwenhoven, J. R. Jude, and G. Knickerbocker. Closed chest cardiac massage. Journal of the American Medical Association (JAMA), pages 84–97, 1960. (Cited on pages 8 and 65.) 144 Bibliography

[109] KTM Advance. Alphega game: Alliance healthcare. "http://www.ktm- advance.com/viewProject_fr.php?id=114", 2010. Accessed 20 Novem- ber 2012. (Cited on page 37.)

[110] Laerdal Medical. MicroSim Prehospital. "http://www.laerdal.com/gb/ docid/12984879/MicroSim-Prehospital", 2012. Accessed 15 Dec 2012. (Cited on pages 69 and 115.)

[111] Laerdal Medical. Laerdal PC SkillReporting System. "http://www.laerdal. com/gb/doc/75/Resusci-Anne-Simulator", 2013. Accessed 18 Feb 2013. (Cited on page 111.)

[112] Laerdal Medical. Resusci Anne Simulator. "http://www.laerdal.com/gb/ doc/75/Resusci-Anne-Simulator", 2013. Accessed 18 Feb 2013. (Cited on pages 14, 111 and 119.)

[113] A. Laikari. Exergaming-gaming for health: A bridge between real world and virtual communities. IEEE 13th International Symposium on Consumer Electronics (ISCE), pages 665–8, 2009. (Cited on pages 35 and 47.)

[114] Lakeside Center for Autisme et Microsoft. Kinetix Academy Begins! "http: //kinetixacademy.co/", 2011. Accessed 20 November 2012. (Cited on pages 39 and 106.)

[115] J. Lansdown and S. Schoﬁeld. Expressive rendering: a review of nonphotoreal- istic techniques. Computer Graphics and Applications, IEEE, 15(3):29–37, 1995. (Cited on page 26.)

[116] Learning Games Lab, New Mexico State University. Science Pirates: The Curse of Brownbeard. "http://www.sciencepirates.org", 2009. Accessed 23 November 2012. (Cited on page 35.)

[117] C. W. Lee. Inﬂuential factors of player loyalty toward online games for achiev- ing commercial success. Australasian Marketing Journal (AMJ), 18(2):81 – 92, 2010. (Cited on page 87.)

[118] S. Lee, J. Kim, J. Kim, and M. Lee. A design of the u-health monitoring system using a nintendo ds game machine. IEEE Engineering in Medicine and Biology Society., pages 1695–8, 2009. (Cited on page 46.)

[119] Less Stress Instructional Services. Adult CPR Simulator 2.0. "http: //www.qweas.com/download/home_education/health_nutrition/ adult_cpr_simulator.htm", 2012. Accessed 31 May 2012. (Cited on page 69.)

[120] F. Liarokapis, L. Macan, G. Malone, G. Rebolledo-Mendez, and S. de Freitas. A pervasive augmented reality serious game. In Conference in Games and Vir- tual Worlds for Serious Applications (VS-GAMES), pages 148 –155, march 2009. (Cited on page 14.) Bibliography 145

[121] I. Liew Suet Fun, M. Sunat, A. Bade, and N. Suaib. Non-photorealistic outdoor scene rendering: techniques and application. In Computer Graphics, Imaging and Visualization, 2004. CGIV 2004. Proceedings. International Conference on, pages 215–220, 2004. (Cited on page 26.)

[122] Lightning Fish Games. NewU: Fitness First Personal Trainer. "http://www. mobygames.com/game/newu-fitness-first-personal-trainer/", 2009. Accessed 23 November 2012. (Cited on page 35.)

[123] J.-K. Lin, P.-H. Cheng, Y. Su, S.-Y. Wang, H.-W. Lin, H.-C. Hou, W.-C. Chiang, S.-W. Wu, J.-J. Luh, and M.-J. Su. Augmented reality serious game framework for rehabilitation with personal health records. In 13th IEEE International Con- ference on e-Health Networking Applications and Services (Healthcom), pages 197 –200, june 2011. (Cited on page 39.)

[124] G. Liu, Z. Jiao, and S. Liu. Tutoring strategy study on game-based experiential learning in vocational school. In Education Technology and Computer Science, 2009. ETCS ’09. First International Workshop on, volume 3, pages 1043–1046, 2009. (Cited on page 115.)

[125] D. Lockery, J. Peters, S. Ramanna, B. Shay, and T. Szturm. Store-and- feedforward adaptive gaming system for hand-ﬁnger motion tracking in telerehabilitation. Information Technology in Biomedicine, IEEE Transactions on, 15 (3):467–473, 2011. (Cited on page 103.)

[126] R. Lopes and R. Bidarra. Adaptivity challenges in games and simulations: A survey. IEEE Transactions on Computational Intelligence and AI in Games, 3:85– 99, June 2011. (Cited on page 29.)

[127] Low D. and Clark N. and Soar J. and Padkin A. and Stoneham A. and Perkins G. and Nolan J. Does use of iresus application on a smart phone improve the performance of an advanced life support provider in a simulated emergency? Resuscitation, 81(2, Supplement):S10 –, 2010. (Cited on page 66.)

[128] Maersk Group. Quest for oil. "http://www.questforoil.com/", 2013. Ac- cessed 20 Aug 2013. (Cited on page 18.)

[129] K. Mania, D. Wooldridge, M. Coxon, and A. Robinson. The effect of visual and interaction ﬁdelity on spatial cognition in immersive virtual environments. Vi- sualization and Computer Graphics, IEEE Transactions on, 12(3):396 –404, may- june 2006. (Cited on pages 11 and 22.)

[130] A. Maslow and R. Frager. Motivation and personality. Harper and Row, 1987. URL http://books.google.es/books?id=L7\_uAAAAMAAJ. (Cited on page 7.) 146 Bibliography

[131] R. McDonnell, M. Breidt, and H. H. Bülthoff. Render me real?: investigating the effect of render style on the perception of animated virtual humans. ACM Trans. Graph., 31(4):91:1–91:11, July 2012. (Cited on pages 11 and 26.)

[132] McGIll University. Grow Your Chi. "http://selfesteemgames.mcgill.ca/ games/chigame.htm", 2006. Accessed 4 January 2013. (Cited on page 32.)

[133] J. A. Mckanna, H. Jimison, and M. Pavel. Divided attention in computer game play: Analysis utilizing unobtrusive health monitoring. 31st Annual Interna- tional Conference of the IEEE EMBS, pages 6247–50, September 2009. (Cited on pages 35 and 46.)

[134] Medical College of Georgia School of Dentistry faculty and students. Sim- ulation helps students learn dental implant procedures. "http://news. georgiahealth.edu/archives/1921", 2009. Accessed 6 March 2012. (Cited on pages 18, 36, 43 and 47.)

[135] Medtronic Foundation. CPR Simulator. "http://www.cprsim.com/", 2012. Accessed 31 May 2012. (Cited on page 69.)

[136] Medtronic Foundation. Save-A-Life Simulator. "http://www. heartrescuenow.com", 2012. Accessed 15 Dec 2012. (Cited on page 66.)

[137] R. Merrill. Introduction to Epidemiology. Jones and Bartlett Publishers, 2010. (Cited on page 44.)

[138] D. MIchael and S. Chen. Serious games: games that educate, train, and inform. Boston MA: Course Technology PTR, 2005. (Cited on pages 7 and 42.)

[139] Microsoft. Introducing Kinect for Xbox 360. "http://www.xbox.com/en-US/ KINECT", 2012. Accessed 26 December 2012. (Cited on pages 10, 27 and 106.)

[140] Microsoft Corporation. Kinect for Windows. "http://www.microsoft. com/en-us/kinectforwindows", 2012. Accessed 22 Feb 2013. (Cited on page 106.)

[141] F.Mili, J. Barr, M. Harris, and L. Pittiglio. Nursing training: 3d game with learning objectives. First International Conference on Advances in Computer-Human Interaction, pages 236–242, 2008. (Cited on page 33.)

[142] J. Miller. MUVE Market Virtual Patient Care Simulation Lab. "http:// www.youtube.com/watch?v=FWUpXar6sh8", 2010. Accessed 7 March 2012. (Cited on page 37.)

[143] Milo Foundation. Project Milo: Game en Platform. "http://milo.hku.nl/", 2012. Accessed 19 November 2012. (Cited on page 41.) Bibliography 147

[144] MIRROR project. 3D Serious games: CLinIC-The Virtual Tutor and Think better CARE-The Virtual Tutor. "http://www.mirror-project.eu/showroom-a- publications/mirror-apps-status/171-3d-serious-games", 2012. Accessed 19 November 2012. (Cited on page 41.)

[145] MIT Teacher Education Program. Outbreak @ MIT. "http://education. mit.edu/ar/oatmit.html", 2004. Accessed 4 January 2013. (Cited on page 31.)

[146] Montreal Science Centre. Sleep: A to Z. "http://www.lesommeil.ca/", 2004. Accessed 4 January 2013. (Cited on page 31.)

[147] S. Moya, S. Grau, D. Tost, R. Campeny, and M. Ruiz. Animation of 3d avatars for rehabilitation of the upper limbs. In Third International Conference on Games and Virtual Worlds for Serious Applications (VS-GAMES), pages 168 –171, may 2011. (Cited on page 39.)

[148] National Safety Training. online safety training. "http://www.nsc.org/ products_training/Training/Pages/OnlineTraining.aspx", 2012. Accessed 30 May 2012. (Cited on page 65.)

[149] H. Nauta and T. Spil. Change your lifestyle or your game is over: The design of a serious game for diabetes. In IEEE 1st International Conference on Serious Games and Applications for Health (SeGAH), pages 1 –7, nov. 2011. (Cited on pages 39 and 47.)

[150] J. Nielsen and R. Molich. Heuristic evaluation of user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’90, pages 249–256, 1990. (Cited on page 118.)

[151] Nike, Inc. Nike + Kinect Training. "http://www.nike.com/us/en_us/c/ training/nike-plus-kinect-training", 2012. Accessed 23 November 2012. (Cited on page 41.)

[152] Nintendo. Brain Age: Train Your Brain in Minutes a Day! "http://www. nintendo.com/games/detail/Y9QLGBWxkmRRzsQEQtvqGqZ63_CjS_9F", 2005. Accessed 4 January 2013. (Cited on page 31.)

[153] Nintendo. WiiFit Plus. "http://wiifit.com", 2012. Accessed 23 November 2012. (Cited on page 32.)

[154] Nintendo EAD. WiiFit. "http://www.nintendo.com/consumer/ gameslist/manuals/Wii_Wii_Fit.pdf", 2007. Accessed 23 November 2012. (Cited on pages 27 and 32.)

[155] Nordic Innovation Centre. MC Urho. "http://nsg.jyu.fi/index.php/MC_ Urho/", 2007. Accessed 26 March 2012. (Cited on page 32.) 148 Bibliography

[156] Nordic Innovation Centre. MoFun Circus. "http://nsg.jyu.fi/index. php/MoFun_Circus", 2007. Accessed 26 March 2012. (Cited on pages 32 and 47.)

[157] K. Nosu, T. Kurokawa, H. Horita, Y. Ohhazama, and H. Takeda. Real time emotion-diagnosis of video game players from their facial expressions and its applications to voice feed-backing to game players. In 2007 International Con- ference on Machine Learning and Cybernetics, volume 4, pages 2208–2212, 2007. (Cited on page 118.)

[158] Novint Technologies Inc. Medical and dental. "http://www.novint.com/ index.php/medicaldental", 2009. Accessed 20 Aug 2013. (Cited on pages 11 and 27.)

[159] Novint Technologies, Inc. The Most Immersive Way to Play Video Games. "http://www.novint.com/index.php/novintfalcon", 2011. Accessed 29 March 2012. (Cited on page 36.)

[160] S. Obdrzalek, G. Kurillo, F. Oﬂi, R. Bajcsy, E. Seto, H. Jimison, and M. Pavel. Accuracy and robustness of kinect pose estimation in the context of coaching of elderly population. In Engineering in Medicine and Biology Society (EMBC), 2012 Annual International Conference of the IEEE, pages 1188 –1193, 28 2012-sept. 1 2012. (Cited on page 106.)

[161] M. Papastergiou. Exploring the potential of computer and video games for health and physical education: A literature review. Computer Education, 53 (3):603 – 622, 2009. (Cited on page 29.)

[162] H. J. Park and S. H. Kim. A bayesian network approach to examining key success factors of mobile games. Journal of Business Research, (0), 2012. (Cited on page 87.)

[163] R. Pepperell. The perception of art and the science of perception. Proc. of SPIE- IS&T Electronic Imaging: Human Vision and Electronic Imaging XVII, 8291, 2012. (Cited on pages 25, 94 and 101.)

[164] G. Pereira, A. Brisson, R. Prada, A. Paiva, F.Bellotti, M. Kravcik, and R. Klamma. Serious games for personal and social learning & ethics: Status and trends. Procedia Computer Science, 15(0):53 – 65, 2012. (Cited on page 17.)

[165] Persuasive Games LLC. Killer Flu. "http://www.persuasivegames.com/ games/game.aspx?game=killerflu", 2009. Accessed 23 November 2012. (Cited on page 35.)

[166] PlayGen. Nanomission. "http://nanomission.org/", 2007. Accessed 20 Aug 2013. (Cited on page 18.) Bibliography 149

[167] Ponder, M., Herbelin, B., Molet, T., Scherteneib, S., Ulicny, B., Papagiannakis, G., Magnenat-Thalmann, N., Thalmann, D. Interactive scenario immersion: Health emergency decision training in just project. Proc. Of 1st International Work- shop on Virtual Reality Rehabilitation, VRMHR2002, VRlab-EPF, pages 87 – 101, November 2002. (Cited on pages 69 and 115.)

[168] PrimeSense, LTD. NITE: Natural interactive middleware. "http://www. primesense.com/solutions/nite-middleware/", 2013. Accessed 22 Feb 2013. (Cited on page 106.)

[169] Public Health Agency of Canada. Buffet busters. "http://www. buffetbusters.ca/", 2011. Accessed 20 November 2012. (Cited on page 39.)

[170] J. Qin, Y.-P.Chui, W.-M. Pang, K.-S. Choi, and P.-A.Heng. Learning blood management in orthopedic surgery through gameplay. IEEE Computer Graphics and Applications, 30(2):45–57, 2010. (Cited on page 115.)

[171] QOVEO. Prevenir la grippe A H1N1. "http://prevenirh1n1.qoveo.com", 2009. Accessed 23 November 2012. (Cited on page 35.)

[172] S. Queiros, J. Vilaca, N. Rodrigues, S. Neves, P. Teixeira, and J. Correia-Pinto. A laparoscopic surgery training interface. In IEEE 1st International Conference on Serious Games and Applications for Health (SeGAH), pages 1–7, nov. 2011. (Cited on page 39.)

[173] RANJ Serious Games. The Great Flu. "http://www.thegreatflu.com", 2009. Accessed 23 November 2012. (Cited on page 35.)

[174] Ray Glennon. White paper, shl group limited,. "http://www.shl.com/ assets/resources/White-Paper-Onboarding.pdf", 2011. Accessed 8 Aug 2013. (Cited on page 17.)

[175] Raylight S.r.l. Train Your Senses. "http://www.raylightgames.com/ games/train-your-senses", 2009. Accessed 23 November 2012. (Cited on page 36.)

[176] M. Reale, P.Liu, L. Yin, and S. Canavan. Art critic: Multisignal vision and speech interaction system in a gaming context. Cybernetics, IEEE Transactions on, PP (99):1–1, 2013. (Cited on page 118.)

[177] Red Hill Studios. Games for people with Parkinson Disease. "http://www. redhillstudios.com/#/projects/games/pdwii/", 2011. Accessed 26 March 2012. (Cited on pages 7, 27, 40 and 46.)

[178] RedCrossOnlineTraining.org. Red Cross online training. "http:// redcrossonlinetraining.org/Distance/Default.aspx", 2012. Ac- cessed 30 May 2012. (Cited on page 65.) 150 Bibliography

[179] P. Rego, P. Moreira, and L. Reis. Serious games for rehabilitation: A survey and a classiﬁcation towards a taxonomy. 5th Iberian Conference on Information Systems and Technologies (CISTI), pages 1–6, June 2010. (Cited on pages 29 and 43.)

[180] Respondesign. Yourself! Fitness. "http://www.respondesign.com/", 2004. Accessed 4 January 2013. (Cited on page 31.)

[181] T. Rhyne. Computer games and scientiﬁc visualization. Communications of the ACM, 45(7):41–44, 2002. (Cited on pages 8 and 87.)

[182] Rick Stirling. Yes, but how many polygons? "http://www.rsart.co. uk/2007/08/27/yes-but-how-many-polygons/", 2007. Accessed 17 Aug 2013. (Cited on page 75.)

[183] J. Riedel and J. Hauge. State of the art of serious games for business and industry. In 17th International Conference on Concurrent Enterprising (ICE), pages 1–8, june 2011. (Cited on page 15.)

[184] D. Roetenberg, P. Slycke, and P. Veltink. Ambulatory position and orientation tracking fusing magnetic and inertial sensing. IEEE Transactions on Biomedical Engineering, 54(5):883–890, 2007. (Cited on page 105.)

[185] M. a. Roubidoux, C. M. Chapman, and M. E. Piontek. Development and evaluation of an interactive web-based breast imaging game for medical students. Academic radiology, 9:1169–78, 2009. (Cited on page 29.)

[186] H. Sabri, B. Cowan, B. Kapralos, M. Porte, D. Backstein, and A. Dubrowskie. Serious games for knee replacement surgery procedure education and training. Procedia - Social and Behavioral Sciences, 2(2):3483 – 3488, 2010. (Cited on page 37.)

[187] SAIC, Inc. OLIVE - On-Line Interactive Virtual Environment. "http://www. saic.com/products/simulation/olive/", 2012. Accessed 7 March 2012. (Cited on pages 41 and 43.)

[188] D. Sandaruwan, N. Kodikara, C. Keppitiyagama, R. Rosa, M. Jayawardena, and P. Samarasinghe. User perception of the physical, behavioral realism of a maritime virtual reality environment. In UKSim 14th International Conference on Computer Modelling and Simulation (UKSim), 2012, pages 172–178, 2012. (Cited on pages 22 and 23.)

[189] B. Sawyer and P.Smith. Serious Games Taxonomy, 2008. Presented at the Game Developers Conference. (Cited on page 42.)

[190] S. Scarle, I. Dunwell, T. Bashford-Rogers, E. Selmanovic, K. Debattista, A. Chalmers, J. Powell, and W. Robertson. Complete motion control of a serious game against obesity in children. Third International Conference on Games Bibliography 151

and Virtual Worlds for Serious Applications, pages 178–179, 2011. (Cited on pages 40 and 46.)

[191] C. SchÃ¶nauer, T. Pintaric, S. J. Kosterink, and M. Vollenbroek. Chronic pain rehabilitation with a serious game using multimodal input. International Con- ference on Virtual Rehabilitation, 2011. (Cited on pages 40, 46 and 106.)

[192] ScienceApplications. Olive: 3D Hospital Training. "http://www.youtube. com/watch?v=MhzD0UO_nUY&feature=related", 2010. Accessed 7 March 2012. (Cited on pages 41 and 43.)

[193] ScienceDaily. Virtual Patient Helps Train Pharmacists Of The Future. "http:// www.danshope.com/news/showarticle.php?article_id=83", 2009. Ac- cessed 7 March 2012. (Cited on page 34.)

[194] Sega. Mind Quiz. "http://www.ubi.com/US/Games/Info.aspx?pId= 5478", 2007. Accessed 4 January 2013. (Cited on page 33.)

[195] F. Semeraro, A. Frisoli, C. Loconsole, F. Banno, G. Tammaro, G. Imbriaco, L. Marchetti, and E. L. Cerchiari. Motion detection technology as a tool for cardiopulmonary resuscitation (cpr) quality training: a randomised crossover mannequin pilot study. Resuscitation, (0):–, 2012. (Cited on pages 69, 106, 109 and 115.)

[196] F.Semeraro, F.Taggi, G. Tammaro, G. Imbriaco, L. Marchetti, and E. L. Cerchiari. icpr: A new application of high-quality cardiopulmonary resuscitation training. Resuscitation, 82(4):436 – 441, 2011. (Cited on page 66.)

[197] Seriousgames.org. The Serious Games Initiative. "http://www. seriousgames.org/", 2002. Accessed 10 July 2013. (Cited on page 7.)

[198] Sim Dietrich. Dot product texture blending and per-pixel lighting. "https://developer.nvidia.com/sites/default/files/akamai/ gamedev/docs/Dot_Product_Texture_Blending.pdf", 2013. Accessed 11 Apr 2013. (Cited on page 19.)

[199] Skills2Learn, Ltd. Skills2Learn Virtual Reality and 3D Simulation Ex- amples. "http://www.skills2learn.com/virtual-reality-case- studies.html", 2010. Accessed 6 March 2012. (Cited on pages 38 and 47.)

[200] A. Sliney and D. Murphy. Jdoc: A serious game for medical learning. First In- ternational Conference on Advances in Computer-Human Interaction, pages 131– 136, February 2008. (Cited on page 33.)

[201] R. Stone, P. Panﬁlov, and V. Shukshunov. Evolution of aerospace simulation: From immersive virtual reality to serious games. In 5th International Conference on Recent Advances in Space Technologies (RAST), pages 655 –662, june 2011. (Cited on page 14.) 152 Bibliography

[202] Stone Meadow Development LLC. CPR & Choking. "https://itunes.apple. com/us/app/cpr-choking/id314907949?mt=8", 2011. Accessed 15 Dec 2012. (Cited on page 66.)

[203] Succubus Interactive. Happy Night Club. "http://www.secrethappynight. com/", 2009. Accessed 23 November 2012. (Cited on page 36.)

[204] I. SuperData Research. US digital games market: January 2013. "http://www. superdataresearch.com/us-digital-games-market", 2013. Accessed 15 Mar 2013. (Cited on page 11.)

[205] C. B. Tanner Olsen, Katelyn Procci. Serious games usability testing: How to ensure proper usability, playability, and effectiveness. In Proceedings First Inter- national Conference of Design User Experience and Usability Theory Methods Tools and Practice (DUXU), pages 625 – 634., 2011. (Cited on page 118.)

[206] R. Tate, J. Haritatos, and S. Cole. Hopelab approach to re-mission. International Journal of Learning and Media, 1:29–35, 2009. (Cited on pages 34, 43 and 47.)

[207] A. Tavassolian, K. Stanley, and C. Gutwin. The effect of temporal adapta- tion granularity and game genre on thetime-balancing abilities of adaptive time-varying minigames. Entertainment Computing, (0):–, 2013. (Cited on page 118.)

[208] The Diablotines. L Affaire BIRMAN. "http://www.gluciweb.com/", 2011. Accessed 20 November 2012. (Cited on page 40.)

[209] The Federation of Public Works RhÃ´ne Alpes. 3d reseaux. "http://www.b2b- games.com/3dreseaux/", 2009. Accessed 26 Jul 2013. (Cited on page 7.)

[210] The Japanese circulation society (JCR). Call and Push simpliﬁed CPR procedure, chest compression and AED. "http://www.j-circ.or.jp/cpr/e- call.html", 2009. Accessed 7 Feb 2013. (Cited on page 107.)

[211] The Partnership for Food Safety Education. The Food Detectives Fight BAC! game. "http://www.fooddetectives.com/", 2008. Accessed 26 March 2012. (Cited on pages 33 and 47.)

[212] Thomson Reuters. Etc simulation. "http://online.wsj.com/article/ HUG1710739.djm.html", 2013. Accessed 26 Jul 2013. (Cited on page 7.)

[213] D. Tost, S. Grau, M. Ferre, P. Garcia, J. Tormos, A. Garcia, and T. Roig. A cognitive telerehabilitation system based on virtual environments. In IEEE Virtual Rehabilitation International Conference, pages 87–93, 2009. (Cited on page 36.)

[214] TruSim, Blitz Games Studios, Ltd. . Triage Trainer. "http://www.trusim. com/", 2008. Accessed 23 November 2012. (Cited on page 38.) Bibliography 153

[215] Ubisoft Divertissements, Inc. Your Shape: Fitness Evolved. "http:// yourshapegame.ubi.com", 2010. Accessed 23 November 2012. (Cited on pages 38 and 106.)

[216] Unity Technologies. Create the games you love with Unity. "http://unity3d. com/unity/", 2012. Accessed 15 January 2012. (Cited on page 75.)

[217] Unity Technologies. Create the games you love with unity. "http://unity3d. com/unity", 2013. Accessed 11 Apr 2013. (Cited on page 93.)

[218] University Medical Center Utrecht. Air Medic Sky 1. "http://www. airmedicsky1.org/", 2012. Accessed 26 March 2012. (Cited on pages 41 and 43.)

[219] A. Vaatanen and J. Leikas. Human-centered design and exercise games. In M. Kankaanranta and P.Neittaanmki, editors, Design and Use of Serious Games, pages 33–47. Springer Science+Business Media, B.V. 2009, 2009. (Cited on pages 36 and 47.)

[220] Vagias, Wade M. Likert-Type Scale Response Anchors. 2006. Clemson Inter- national Institute for Tourism & Research Development, Department of Parks, Recreation and Tourism Management. Clemson University. (Cited on pages 98, 121 and 124.)

[221] E. van Loon, C. Peper, A. van de Rijt, and A. Salverda. Serious gaming to improve bimanual coordination in children with spastic cerebral palsy. In International Conference on Virtual Rehabilitation (ICVR), pages 1 –2, june 2011. (Cited on pages 40 and 45.)

[222] M. Vazquez, V. Santana-Lopez, M. Skodova, J. Ferrero-Alvarez-Rementeria, and A. Torres-Olivera. Hand hygiene training through a serious game: New ways of improving safe practices. In 2011 IEEE 1st International Conference on Serious Games and Applications for Health (SeGAH), pages 1 –2, nov. 2011. (Cited on pages 40 and 47.)

[223] M. L. Verduin, S. D. LaRowe, H. Myrick, J. Cannon-Bowers, and C. Bowers. Computer simulation games as an adjunct for treatment in male veterans with alcohol use disorder. Journal of Substance Abuse Treatment, (0), 2012. (Cited on page 41.)

[224] Vermont department of health. Khemia. "http://www.vtquitnetwork.org/ game", 2008. Accessed 23 November 2012. (Cited on page 33.)

[225] A. Vidani, L. Chittaro, and E. Carchietti. Assessing nurses acceptance of a serious game for emergency medical services. In Games and Virtual Worlds for Serious Applications (VS-GAMES), 2010 Second International Conference on, pages 101– 108, 2010. (Cited on page 115.) 154 Bibliography

[226] A. Vidani, L. Chittaro, and E. Carchietti. Assessing nurses acceptance of a serious game for emergency medical services. Second International Conference on Games and Virtual Worlds for Serious Applications (VS-GAMES), pages 101–108, March 2010. (Cited on pages 38, 43 and 47.)

[227] Virtual Heroes, Inc. Zero Hour: America Medic. "http://virtualheroes. com/projects/zero-hour-americas-medic", 2009. Accessed 29 January 2012. (Cited on page 36.)

[228] Visual Imagination Software. Chirurgie Simulator. "http://www.vis-games. de/details_chirurgie.html", 2010. Accessed 23 November 2012. (Cited on page 38.)

[229] VSTEP. Nautis. "http://www.nautissim.com/", 2009. Accessed 26 Jul 2013. (Cited on page 7.)

[230] VSTEP. Rescuesim. "http://www.rescuesim.com", 2013. Accessed 20 Aug 2013. (Cited on page 18.)

[231] M. Waite and S. Hawker. Oxford Paperback Dictionary and Thesaurus. Oxford Paperbacks. Oxford University Press, 2009. (Cited on page 29.)

[232] A. Wang, T. fsdahl, and O. Morch-Storstein. An evaluation of a mobile game concept for lectures. In Software Engineering Education and Training, 2008. CSEET ’08. IEEE 21st Conference on, pages 197–204, 2008. (Cited on page 118.)

[233] Q. Wang, O. Sourina, and M. K. Nguyen. Eeg-based serious games design for medical applications. International Conference on Cyberworlds (CW), pages 270 –276, 2010. (Cited on pages 38, 45 and 46.)

[234] Warner Bros. Entertainment Inc. Pamoja Mtaani: The partnership for an HIV- free Generation . "http://hivfreegeneration.warnerbros.com", 2008. Accessed 23 November 2012. (Cited on page 33.)

[235] C. Watters, S. Oore, M. Shepherd, A. Abouzied, A. Cox, M. Kellar, H. Kharrazi, F. Liu, and A. Otley. Extending the use of games in health care. 39th Annual Hawaii International Conference on System Sciences (HICSS), page 88b, January 2006. (Cited on page 29.)

[236] Wikipedia. Direct instruction. "http://en.wikipedia.org/wiki/Direct_ instruction", 2013. Accessed 11 Sep 2013. (Cited on page 116.)

[237] A. Williams. Bulldog pushes to demonstrate fourth generation haptics. "http://www.electronicsweekly.com/electro-ramblings/latest- news/bulldog-pushes-to-demonstrate-fourth-generation- haptics-2013-06/", 2013. Accessed 16 Sep 2013. (Cited on page 27.) Bibliography 155

[238] World Health Organization. Constitution of the world health organization - basic documents, forty-ﬁfth edition. Supplement, October 2006. (Cited on page 29.)

[239] S. Wright. Chapter 6 - rotoscoping. In Compositing Visual Effects, pages 101 – 114. Focal Press, Boston, 2008. (Cited on page 25.)

[240] J. Yan, Y. Li, E. Zheng, and Y. Liu. An accelerated human motion tracking system based on voxel reconstruction under complex environments. In Computer Vision, ACCV 2009, volume 5995 of Lecture Notes in Computer Science, pages 313–324. Springer Berlin Heidelberg, 2010. (Cited on page 104.)

[241] H. L. Yi. Designing serious games to enhance political efﬁcacy and critical thinking disposition for college students: The case of taiwan. In Third International Conference on Games and Virtual Worlds for Serious Applications (VS-GAMES), pages 148 –151, may 2011. (Cited on page 15.)

[242] M. H. Zadeh. Advances in Haptics. Online; InTech, 722 pages, 2010. (Cited on pages 27 and 103.)

[243] F. Zhou, H.-L. Duh, and M. Billinghurst. Trends in augmented reality tracking, interaction and display: A review of ten years of ismar. In 7th IEEE/ACM Inter- national Symposium on Mixed and Augmented Reality (ISMAR), pages 193 –202, Sep. 2008. (Cited on page 15.)

[244] H. Zhou and H. Hu. Human motion tracking for rehabilitation, a survey. Biomed- ical Signal Processing and Control, 3(1):1 – 18, 2008. (Cited on pages 15, 105, 107 and 108.)

[245] Zumba Fitness LLC. Zumba Fitness. "http://www.zumba.com", 2012. Ac- cessed 23 November 2012. (Cited on pages 38 and 106.)

[246] M. Zyda. From visual simulation to virtual reality to games. Computer, 38(9): 25 – 32, sept. 2005. (Cited on pages 7 and 14.)

LISSALIFE SUPPORT SIMULATION APPLICATION