Hamilton Institute Multimodal, Embodied and Location-Aware Interaction A dissertation submitted for the degree of Doctor of Philosophy by Steven Strachan, BSc., MSc. Supervisor: Roderick Murray-Smith Hamilton Institute National University of Ireland, Maynooth Ollscoil na hEireann,¶ M¶aNuad January 2007 Contents Title Page . i Table of Contents . ii Contributing Publications . viii Acknowledgments . x 1 Introduction 1 1.1 Designing Interaction . 1 1.2 Instrumenting Interaction . 2 1.3 Computer-Human Interaction . 3 1.3.1 Input . 4 1.3.2 Novel Interaction . 6 1.3.3 Embodied Interaction . 6 1.4 Spatial Interfaces . 9 1.5 Embracing Uncertainty . 11 1.6 Modelling Interaction . 12 1.7 Thesis Outline . 13 1.7.1 Our Place . 13 1.7.2 Structure . 14 1.8 Thesis Claims . 15 2 Background and Challenges 17 2.1 Introduction . 17 2.2 Inertial Sensing . 18 2.3 Our Sensors . 20 2.3.1 Hardware . 20 2.3.2 Other Hardware . 21 2.4 Mobile Signals . 22 2.5 Mobile Limitations . 24 2.6 Control Theory . 26 2.6.1 Manual Control . 28 Discrete or Continuous Control . 29 Tracking . 30 2.7 Inferring Intention . 30 2.7.1 Uncertainty . 31 2.7.2 Constraints . 32 2.8 Feedback . 33 ii Contents 2.8.1 Multiple-Modalities . 34 2.9 Sensor Fusion . 36 2.10 Location-Aware Technologies . 37 2.11 Location-Aware Applications . 40 2.12 Location-Aware Audio . 41 2.13 Gesture Recognition . 43 2.14 Gesture Variability . 45 2.15 Gesture Controlled Applications . 46 2.16 Social Issues . 48 3 Bodyspace 49 3.1 Summary . 49 3.2 Introduction . 49 3.3 Gesture Recognition and Segmentation . 52 3.3.1 Our Approach . 53 Segmentation . 53 Recognition . 56 Training . 58 3.4 Utilising Constraints . 59 3.4.1 Varying Planes . 62 3.5 Isomorphism . 63 3.6 Approaches to Feedback . 64 3.6.1 Dynamic Movement Primitives . 65 Movement Primitive Example . 66 Muscle Tremor . 67 3.7 BodyMusic: Gesture Controlled MP3 Player . 69 3.7.1 Model: Ball in Bowl . 70 3.7.2 System Testing . 76 Method . 76 Results . 76 Observations . 79 3.8 Other Potential Applications . 84 3.8.1 O®-The-Wall Interaction . 84 3.9 Discussion and Conclusions . 88 4 Whereable Computing 90 4.1 Summary . 90 4.2 Introduction . 90 4.3 Monte Carlo Propagation For Browsing Virtual Environments 92 4.3.1 Likelihood Maps . 94 4.3.2 A Priori Sensor Uncertainty Maps . 96 4.3.3 Variable Time Horizon Prediction . 98 4.4 gpsTunes: Navigation By Audio . 99 4.4.1 Non Probabilistic Approach . 99 4.4.2 Probabilistic Approach . 101 4.5 Target Acquisition . 101 iii Contents 4.5.1 Granular Synthesis . 101 4.5.2 Outdoor Field Trial . 102 Method . 102 Results . 104 Comments from Participants . 105 Observations . 105 4.5.3 In A Virtual Environment . 106 Experimental Details . 107 Results . 108 4.5.4 Discussion . 109 4.6 Trajectory Following . 111 4.6.1 Our Application . 114 4.6.2 Feedback . 116 4.7 Forming Trajectories . 116 4.8 Trajectory Following Trials . 117 4.8.1 Method . 118 4.8.2 Results . 118 4.9 Human Behaviour Modelling . 127 4.9.1 Our Model . 129 4.9.2 Model Testing . 131 Recreating Behaviour . 131 4.9.3 Improvements . 134 4.10 Discussion and Conclusions . 135 5 Messages in the Air 137 5.1 Summary . 137 5.2 Introduction . 137 5.3 Augmented Reality and Virtual Environments . 138 5.4 Contextual Interaction . 140 5.5 Density Exploration . 141 5.5.1 airMessages . 142 Location Projection . 144 5.5.2 System Testing . 145 Method . 146 5.5.3 Results and Observations . 148 5.6 Potential Applications . 161 5.6.1 Social Networking . 161 5.6.2 Geographical Blogging . 163 5.6.3 Demographics . 163 5.6.4 Tourism . 164 5.6.5 File Sharing . 164 5.6.6 Sculpting the Virtual Environment . 165 5.7 Discussion and Conclusions . 165 iv Contents 6 Conclusions 167 6.1 Theoretical Framework . 167 6.2 BodySpace . 168 6.3 Whereable Computing . 169 6.4 Outlook . 170 A The Global Positioning System 172 A.1 Navigation . 172 A.2 Accuracy . 173 A.3 Sources Of Error . 174 A.3.1 Satellite Geometry . 174 A.3.2 Signal Shadowing . 175 A.3.3 Atmospheric E®ects . 176 A.3.4 Ephemeris and clock errors . 176 A.3.5 Multipath E®ects . 177 A.4 Other Applications . 178 B Inertial Sensing For Mobile Devices 179 B.1 Introduction . 179 B.2 Mobile Movement . 180 B.3 Coordinate Systems . 180 B.3.1 Navigation Equation . 182 B.4 Sensors . 184 B.4.1 MEMS inertial sensors . 184 B.4.2 Accelerometers . 185 B.4.3 Gyroscopes . 185 B.4.4 Magnetometers . 186 B.5 Sensor Output . 188 B.5.1 Sensor Placement . 189 Rigid Body Kinematics . ..