A Person-Centric Design Framework for At-Home Motor Learning in Serious Games by Ramin Tadayon A Dissertation Presented in Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy Approved July 2017 by the Graduate Supervisory Committee: Sethuraman Panchanathan, Chair Troy McDaniel Ashish Amresh Arthur Glenberg Baoxin Li ARIZONA STATE UNIVERSITY August 2017 ABSTRACT In motor learning, real-time multi-modal feedback is a critical element in guided training. Serious games have been introduced as a platform for at-home motor training due to their highly interactive and multi-modal nature. This dissertation explores the design of a multimodal environment for at-home training in which an autonomous system observes and guides the user in the place of a live trainer, providing real-time assessment, feedback and difficulty adaptation as the subject masters a motor skill. After an in-depth review of the latest solutions in this field, this dissertation proposes a person-centric approach to the design of this environment, in contrast to the standard techniques implemented in related work, to address many of the limitations of these approaches. The unique advantages and restrictions of this approach are presented in the form of a case study in which a system entitled the "Autonomous Training Assistant" consisting of both hardware and software for guided at-home motor learning is designed and adapted for a specific individual and trainer. In this work, the design of an autonomous motor learning environment is approached from three areas: motor assessment, multimodal feedback, and serious game design. For motor assessment, a 3-dimensional assessment framework is proposed which comprises of 2 spatial (posture, progression) and 1 temporal (pacing) domains of real- time motor assessment. For multimodal feedback, a rod-shaped device called the "Intelligent Stick" is combined with an audio-visual interface to provide feedback to the subject in three domains (audio, visual, haptic). Feedback domains are mapped to modalities and feedback is provided whenever the user's performance deviates from the ideal performance level by an adaptive threshold. Approaches for multi-modal integration i and feedback fading are discussed. Finally, a novel approach for stealth adaptation in serious game design is presented. This approach allows serious games to incorporate motor tasks in a more natural way, facilitating self-assessment by the subject. An evaluation of three different stealth adaptation approaches are presented and evaluated using the flow-state ratio metric. The dissertation concludes with directions for future work in the integration of stealth adaptation techniques across the field of exergames. ii TABLE OF CONTENTS Page LIST OF TABLES ................................................................................................................ viii LIST OF FIGURES ................................................................................................................. ix CHAPTER 1 INTRODUCTION ...................................................................................................... 1 1.1 Dissertation Overview ..................................................................... 1 1.2 Previously Published Work ............................................................. 5 2 SYSTEMS OVERVIEW ................... ....................................................................... 6 2.1 Virtual Reality/Augmented Reality ................................................. 6 2.2 Commercial Game Hardware .......................................................... 8 2.3 Serious Games for Motor Learning ............................................... 10 2.4 Player Interest Considerations ....................................................... 14 2.4.1 Motivational Factors ......................................................... 14 2.4.2 Experiential Factors .......................................................... 15 2.4.3 Demographic Factors ........................................................ 17 2.4.4 Cognitive Factors .............................................................. 18 2.4.5 Design for Adaptation ....................................................... 18 2.5 Affective Game Design ................................................................. 19 3 MOTOR LEARNING OVERVIEW ........................................................................ 23 3.1 Cognitive Models ........................................................................... 23 3.2 Mirror Neuron System Overview .................................................. 27 3.3 Mirroring Mechanisms and Design Considerations ..................... 29 iii CHAPTER Page 3.3.1 Associative Account ......................................................... 29 3.3.2 Predictive Coding Account ............................................... 34 4 CASE STUDY: AUTONOMOUS TRAINING................ ..................................... 38 4.1 Challenges and Objectives ............................................................. 38 4.2 Autonomous Training Assistant Overview ................................... 38 4.2.1 System Design .................................................................. 39 4.2.2 Intelligent Stick ................................................................. 40 4.2.3 Authoring Software........................................................... 41 4.2.4 Virtual Training Software ................................................. 45 4.3 Evaluation 1: Usability Study ........................................................ 46 4.3.1 Procedure........................................................................... 47 4.3.2 Results and Discussion ..................................................... 49 4.4 Case Study Overview..................................................................... 50 4.5 Single Subject vs. Group Assessment .......................................... 51 5 MOTOR ASSESSMENT ................ ........................................................................ 56 5.1 Overview of Assessment Mechanisms .......................................... 56 5.2 Evaluation 2: Assessment Model .................................................. 57 5.2.1 Procedure........................................................................... 57 5.2.2 Results and Discussion ..................................................... 60 5.3 Evaluation 3: Unsupervised (At-Home) Assessment ................... 62 5.3.1 Procedure........................................................................... 62 5.3.2 Results and Discussion ..................................................... 63 iv CHAPTER Page 6 MOTOR FEEDBACK ............................................................................................. 65 6.1 Overview of Feedback Mechanisms ............................................. 65 6.2 Multimodal Mapping Model ......................................................... 67 6.3 Haptic Guidance Considerations ................................................... 73 6.3.1 Continuous Trajectory Representation ............................. 75 6.3.2 Tactile Tethering ............................................................... 76 6.3.3 Volumetric Error Augmentation ...................................... 79 6.4 Fine Postural Correction ................................................................ 81 6.5 Information Transfer Overview ..................................................... 84 6.5.1 Perceptual Bandwidth Variations ..................................... 84 6.5.2 Information Transfer ......................................................... 86 6.5.3 Design Implications .......................................................... 87 6.6 Perceptual Bandwidth Augmentation ............................................ 92 6.6.1 Haptic Information ............................................................ 93 6.6.2 Audio Information ............................................................ 93 6.6.3 Visual Information ............................................................ 95 6.6.4 Multimodality ................................................................... 96 6.7 Feedback Frequency and Fading ................................................... 97 6.7.1 Motor Task Complexity .................................................... 99 6.7.2 Phases of Learner Proficiency ........................................ 100 6.7.3 Feedback Timing and Frequency ................................... 103 6.7.4 Feedback Delivery Strategy ........................................... 103 v CHAPTER Page 6.8 Tolerance Threshold Adaptation ................................................ 106 6.9 Multimodal Fusion Techniques .................................................. 112 6.9.1 Style 1: Alternate ........................................................... 113 6.9.2 Style 2: Exclusive ........................................................... 114 6.9.3 Style 3: Synergistic ......................................................... 116 6.9.4 Style 4: Concurrent ......................................................... 117 6.9.5 Discussion ....................................................................... 118 6.10 Mode Prioritization ................................................................... 119 6.11 Multimodal Conflict Resolution ............................................... 122 6.12 Evaluation 4: Multimodal Feedback ........................................ 124 6.12.1 Procedure .....................................................................