JIRC - Orléans Mercredi 29 Mai 2013
Laurence Nigay EHCI group
Interaction modality “EHCI” for and Engineering for Human-Computer Interaction Created in 1990 multimodality 9 professors and assistant professors 1 CNRS researcher Joseph Fourier University 1 CNRS research engineer part time LIG Grenoble Informatics Laboratory 14 PhD students 5 project engineers 1 visiting researcher Engineering for 10 master students Human-Computer Interaction
JIRC Journées Informatique de la Région Centre
EHCI group EHCI group: Research thematic
EHCI group belongs to LIG Software Engineering for HCI (Computer Science Laboratory of Grenoble) 188 academic researchers Interaction 367 doctoral / post-doctoral students, visitors, engineers Computer science Supported by 45 technical and administrative staff contribution System Human Human sciences
23 autonomous research teams Understanding the human-computer phenomena Establishing links between psychology-ergonomics 4 principal themes and software engineering Infrastructure (networks and data) Software (foundations and design models) Designing, developing and evaluating interaction techniques Interaction (perception, action and dialog) Knowledge (learning, agent models and web-ontologies) Developing conceptual and technical tools based on HCI principles: Utility, Usability, Context
Laboratory of excellence EHCI group: Research axes Persyval-lab
5 complementary research axes Pervasive Systems & Algorithms at the convergence of the physical and digital worlds
Multimodal interaction Interaction with small handheld devices Signal & Information & Mixed reality interaction Automatic Communication User interface plasticity Control Technology marrying New interaction techniques physicality & computation
Mathematics and Simulation
Laurence Nigay - [email protected] 1 JIRC - Orléans Mercredi 29 Mai 2013
Scientific themes: Outline Design trade-offs
Research framework Designers face a fundamental Scientific themes challenge: Research approach How to balance the trade-off HCI & Ubiquitous computing between Three research axes power of expression and Foundations of my work simplicity of interaction? Interaction modality Multimodality Combining the real and virtual worlds
© W. Mackay
Scientific themes: Scientific approach: Design trade-offs deduction and induction
Theorical Model New Revised model model Research challenge? Move the curve!
Empirical Observation Evaluation Re- evaluation
© W. Mackay © W. Mackay
Scientific approach: deduction and induction HCI & Ubiquitous Computing
Ubiquitous computing 1991 Model New Revised Theorical M. Weiser Scientific American model model Calm technology
Artifact Prototype System design Invisible technology Technology available at any place Symbiosis of the real and digital worlds
Empirical Observation Evaluation Re- evaluation
© W. Mackay
Laurence Nigay - [email protected] 2 JIRC - Orléans Mercredi 29 Mai 2013
HCI & Ubiquitous Computing HCI & Ubiquitous Computing
Virtual Embodied Reality Virtuality
Ubiquitous Computing HCI & Ubiquitous Computing
Three revolutions in computing HCI in the context of 1010 MainframeUbiquitousPC Computing Computing Computing “Ubiquitous computing” 109 1N 1computer computerscomputer a seamless interactive environment 108 N users1 user unobtrusive, everywhere 107 often invisible and yet in our consciousness 106 105 104 103 102 101
1940 1950 1960 1970 1980 1990 2000 2010 Machines sold per year (M. Weiser, UIST 94)
HCI & Ubiquitous Computing HCI & Ubiquitous Computing
HCI in the context of HCI in the context of “Ubiquitous computing” “Ubiquitous computing”
“Our surrounding is the interface” Unprecedented challenges for interaction to a universe of integrated services. This will enable citizens to access IST services wherever they are, design whenever they want, and in the form that is most “natural” Combining the real and virtual worlds for them Multiple interaction devices/modalities Small and large interaction surfaces Dynamic contexts of use
Laurence Nigay - [email protected] 3 JIRC - Orléans Mercredi 29 Mai 2013
HCI & Ubiquitous Computing HCI & Ubiquitous Computing
HCI in the context of My scientific goals: “Ubiquitous computing” 1. Design and realisation of interaction techniques Unprecedented challenges for interaction for large information spaces design (interactive visualisation) … while defining an opportunity for the users’ 2. Design and realisation of acceptance of innovative interaction techniques interaction techniques on mobile devices going beyond the desktop paradigm (mobile techniques) 3. Design and realisation of The Xerox Star has reached its limits interaction techniques for combining the real and virtual worlds (combining real and virtual worlds)
HCI & Ubiquitous Computing Outline
My scientific goals: Research framework Scientific themes Research approach 1. Design and realisation of Ubiquitous computing interaction techniques for large information spaces Three research axes (visualisation) From the point of view of Foundations of my work 2. Design and realisation of Interaction modality interaction techniques on mobile devices and Interaction modality (mobile techniques) Multimodality 3. Design and realisation of Multimodality interaction techniques Combining the real and virtual worlds for combining the real and virtual worlds (combining real and virtual worlds)
Interaction modality Interaction modality
Human Representational Modality = (device, interaction language) Theory ICS acoustic Subsystems articulatory ears mouth subsystem subsystem APU Cambridge A set of sensors (input devices)
or effectors (output devices) face limb & visual tactile ICS as predicting retina subsystem body subsystems A processing facility based on a language cognitive hand resources Internal P1 camera Input modality Output modality Digital involved in using Processes P5 P2 and choosing screen pen
P3 modalities loud microphone speaker
P4 keyboard mouse touch screen
Laurence Nigay - [email protected] 4 JIRC - Orléans Mercredi 29 Mai 2013
Interaction modality Interaction modality
Modality = (device, interaction language) Recent interaction paradigms such as perceptual UI tangible UI and embodied UI open a vast world of possibilities
M1 = (microphone, natural language) M2 = (keyboard, command language) M3 = (mouse, direct manipulation) M4 = (PDA, 3D gesture) embodied UI M5 = (HMD, 3D graphics) AR M6 = (bottle-sensor, 3D gesture) tangible UI M7 = (GPS, localization) perceptual UI M8 = (Tongue display, 2D shape)
Interaction modality Interaction modality
INPUT Modality =
Modality = < , natural language>
Flights from M =
M =
Interaction modality Interaction modality
Input M =
Perceptual browser project Gesture Recognition
SHAKE triple axis accelerometer, two capacitive sensors and a vibrotactile actuator
Laurence Nigay - [email protected] 5 JIRC - Orléans Mercredi 29 Mai 2013
Interaction modality Definition of a modality
Input M =
Tokens
The position of tokens are tracked by a vision based mechanism. A tracking video camera is fixed on the top of the table. NAVRNA: A system to visualize, explore and edit RNA.
Definition of a modality Definition of a modality
Input M =
Tertiary structure
Secondary structure a) One token: panning action
b) Two tokens: zoom + rotate
Definition of a modality Definition of a modality
Input M = Input M =
Two-handed interaction: Bending one branch of RNA
Laurence Nigay - [email protected] 6 JIRC - Orléans Mercredi 29 Mai 2013
Definition of a modality Definition of a modality
Input M = Input M =
15 degrees of freedom Thumb: 5 degrees of freedom
Bulchholz, B., Armstrong, T. J., (1992) A Kinematic model of the human hand to evaluate its prehensile capabilities, J. Biomecanics.
Definition of a modality Definition of a modality
Input M =
Definition of a modality Definition of a modality
Input M = Input M =
Laurence Nigay - [email protected] 7 JIRC - Orléans Mercredi 29 Mai 2013
Definition of a modality Definition of a modality
Input M = Input M =
Definition of a modality Definition of a modality
Two handed-interaction Input M = Wavelet menu on iPhone
Space on screen No keyboard for shortcuts (novice mode) The best way to interact is to use only one-hand Eye-free interaction
Input modalities on small devices Input modalities on small devices
Marking menus Marking menus Multi-stroke marking menus • Advantages Instead of considering a spatial compound stroke, – Circular design Multi-Stroke menus introduce a serie of simple strokes – Fluid transition – Scale independance Require less physical input space in novice & expert modes A submenu is displayed on top of • Limitations its parent menu – Screen space requirement Overlapped marks – Number of commands Increase accuracy in expert mode • Error rate in expert mode • Ambiguous marks in expert mode Increase the number of items No ambiguous gestures in expert mode
Laurence Nigay - [email protected] 8 JIRC - Orléans Mercredi 29 Mai 2013
Input modalities on small devices Input modalities on small devices
Wave menus
Wave menus Few physical input space A submenu is displayed at the center of the menu system. In order to remain visible, its parent menu is then enlarged to Novice Mode surround this submenu In expert mode, Wave Menus (1) work exactly the same way as Multi-Stroke Menus (2) Expert Mode
Input modalities on small devices Input modalities on small devices
Wavelet menus: extension on Iphone Wavelet menus: extension on Iphone
• The Wavelet menu appears centered around the contact point. • By drawing a stroke towards the desired item, the first level is enlarged permitting progressive appearance of the submenu. • A second stroke selects an item in the submenu.
Input modalities on small devices Input modalities on small devices
Wavelet menus: Wavelet menus: extension on Iphone
Long lists management: the linear list appears in the center of the Wavelet menu and is surrounded by its parent menus.
Laurence Nigay - [email protected] 9 JIRC - Orléans Mercredi 29 Mai 2013
Input modalities on small devices Input modalities on small devices
LeafMenu: extension of linear menus LeafMenu: extension of linear menus
Definition of a modality Definition of a modality
Input M =
Object-Sensor = Mixed objects Ambient room MIM: Model of Mixed Interaction Media Lab MIT A conceptual model for designing augmented objects OP (Object Prototyping): A toolkit based on the Input M =
Movements Pressure Torsion
Definition of a modality Definition of a modality
Input M =
mixed object Toolkit developed in Qt physical digital Toolkit that includes ARToolkit, Phidgets, properties Linking properties modalities Interface-Z sensors
OP - Prototyping mixed objects: Combined development of form and interaction.
Scott E. Hudson HCI CMU
Laurence Nigay - [email protected] 10 JIRC - Orléans Mercredi 29 Mai 2013
Definition of a modality Definition of a modality
Input M =
Definition of a modality Definition of a modality
Input Modalities (sensing modalities) Input Modalities M1 =
Get M1 = (micro, command) M2 = (trackpad, 2D gesture) M3 = (camera, 3D gesture)
TROC project
Definition of a modality Definition of a modality
OUTPUT Modality =
CASPER project M =
Laurence Nigay - [email protected] 11 JIRC - Orléans Mercredi 29 Mai 2013
Outline Multimodality
Research framework Design space Set of atomic/combined Scientific themes modalities Research approach Combination Ubiquitous computing Context Modality
Three research axes of modalities Foundations of my work Interaction modality Information Selection of Multimodal Multimodality to be one or several modalities Expression conveyed Augmented Reality Actor of the selection
Selection criteria
Multimodality Multimodality
Who is performing the selection Who is performing the selection
No adaptation Designer User System Adaptability
Information Selection of Multimodal to be one or several modalities Expression Adaptivity Selection by Selection by Selection by conveyed the designer the user the system Actor of the selection
Multimodality Multimodality Adaptability Adaptability
Selection of the modalities by the user Wizard of oz
Go to the middle of the message
Gestural modality Speech
Accomplice Subject
Embodied modality Direct manipulation
Laurence Nigay - [email protected] 12 JIRC - Orléans Mercredi 29 Mai 2013
Multimodality Multimodality Adaptability Adaptability
Use of the modalities Selection of the modalities by the user All sessions / All subjects
Speech M1 = (trackpad, 2D gesture) Direct manipulation Tactile Gesture
Embodied M2 = (camera, 3D gesture) Gestural
The subjects used all of the modalities Individual preferences leading in some cases to Get Speech specialization M3 = (micro, command)
Multimodality Multimodality Adaptability Adaptivity
Use of the modalities Selection of the modalities by the system All sessions / All subjects Context-aware systems
24% Tactile 45% Gestural Speech 31% Tata tata ta ta ta taaaa tata
Individual preferences leading in some cases to specialization Subject 4: Tactile modality for rotating a puzzle piece Subject 9: Speech for rotating and collecting a puzzle piece
Multimodality Adaptivity Multimodality
Selection of the modalities by the system Design space Context-aware systems Grand challenges To what extent can modality selection & Combination Context Modality configuration be automated? of modalities Proactiveness (take the initiative), Predictability, Transparency Where full automation is not possible or desirable, how can human involvement be Information Selection of Multimodal to be one or several modalities Expression supported? conveyed End-user programming of context-sensitive environments Actor of the selection Selection criteria
Laurence Nigay - [email protected] 13 JIRC - Orléans Mercredi 29 Mai 2013
Multimodality Multimodality
Selection criteria: Context Design space Set of atomic/combined modalities Type of infor. Combination Context Modality of modalities Temporality
Dialogue Dialogue Information Selection of Multimodal Interface Interface Persistent to be one or several modalities Expression conveyed User User Transient Physical env. Physical env. Variability Selection criteria Static Dynamic
Atomic and combined modalities Atomic and combined modalities
Interaction modelling at the modality level A vast world of atomic and combined modalities Rich enough to express differences any physical object can be involved in interaction as a device
We can no longer expect to model each input and output modality in all their diversity at the Abstract enough to enable reasoning concrete level among modalities (vast world of modalities) We need to reason about modalities at a higher any physical object can be involved in interaction level of abstraction between modality and task/system/service/ context issues
Multimodality Multimodality
Characterisation of a modality Characterisation of a modality
Active modalities Human sense For inputs, active modalities are used by the user to Spatial Dimension: 1D 2D ... issue a command to the computer such as a pedal to Location O. Bernsen 93 move a laparoscope in a CAS system Temporal Linguistic Transient/Persistent Analogue Active / Passive Arbitrary Passive - Implicit modalities Modality Private / Public Passive modalities are used to capture relevant information for enhancing the realization of the task, information that is not explicitly expressed by the user Physical level Logical level to the computer (PUI). For example tracking position Modality =
Laurence Nigay - [email protected] 14 JIRC - Orléans Mercredi 29 Mai 2013
Multimodality Multimodality
Characterisation of a modality Characterisation of a modality
Physical level Physical level Human sense: Sight Human sense: Sight Spatial: Location = screen Spatial: Temporal: Persistent Location = operating field Public Temporal: Persistent Logical level Private 2D Logical level Non Analogue 3D Arbitrary Analogue Non arbitrary
Multimodality Multimodality
Characterisation of a modality Design space Set of atomic/combined modalities
Combination Physical level Context Modality Human sense: Sight of modalities Spatial: Location = screen Temporal: Persistent Public Logical level Information Selection of Multimodal 3D to be one or several modalities Expression Analogue conveyed Non arbitrary
Multimodality Multimodality
Combination of modalities Tasks Several studies UOM 94 / TYCOON 95 / CARE 95
CARE properties Relationships between Devices, Interaction Languages languages and Tasks C : Complementarity “Put that there” paradigm A : Assignment Devices R. Bolt, MIT, 1980 R : Redundancy E : Equivalence
Laurence Nigay - [email protected] 15 JIRC - Orléans Mercredi 29 Mai 2013
Multimodality Multimodality
Flight simulator of a military fighter, used for studying future interaction techniques in the cockpit
Modalities: M1: Aircraft location M2: Pilot’s orientation M3: HOTAS commands M4: Speech commands
Multimodality Multimodality
TROC: a game based on the technique of barter M1 = (Magnetometer, orientation) M2 = (GPS, location) Complementarity of M1 and M2 for selecting an object Complementarity 2
3D orientation (radians)
Magnetometer
Multimodality Multimodality
CARE properties Combination of modalities Devices Languages Tasks D L a set D of Devices can a set L of Languages can be : be : - equivalent - equivalent CARE properties - redundant - redundant - complementary - complementary according to according to
a particular language li a particular task ti a device d can be : i assigned to a language lj can be : New combination space assigned to Different schemas and aspects of Modality M Tasks combinations M a set M of modalities can be : 5 aspects: temporal, spatial, articulatory - equivalent - redundant - complementary syntactic and semantic according to a particular task t i 5 schemas: [Allen 83] a modality m j can be : assigned to
Laurence Nigay - [email protected] 16 JIRC - Orléans Mercredi 29 Mai 2013
Multimodality Multimodality
Combination of modalities AR Puzzle
Combination schemas
Temporal Anachronism Sequence Concomitance Coincidence Parallelism
Spatial Separation Adjacency Intersection Overlaid Collocation
Articulatory Independence Fission Fission Partial Total Duplication Duplication Duplication Syntactic Difference Completion Divergence Extension Twin
Semantic Concurrency Complementarity Complementarity Partial Total & Redundancy Redundancy Redundancy Combination aspects
Turn 48° to the left
Multimodality Multimodality
PERM system: a CAS system for kidney Puzzle puncture
M1 =
M2 =
Multimodality Multimodality
Combination of Design space Set of atomic/combined M2 =
of modalities
Temporal Anachronism Sequence Concomitance Coincidence Parallelism
Spatial Separation Adjacency Intersection Overlaid Collocation Information Selection of Multimodal to be Articulatory Independence Fission Fission Partial Total one or several modalities Expression Duplication Duplication Duplication conveyed Syntactic Difference Completion Divergence Extension Twin
Semantic Concurrency Complementarity Complementarity Partial Total & Redundancy Redundancy Redundancy Combination aspects
Laurence Nigay - [email protected] 17 JIRC - Orléans Mercredi 29 Mai 2013
Combining Outline the real and virtual worlds
Research framework Scientific themes Research approach Non-tangible world Ubiquitous computing Real world Computer world Three research axes Virtual world Foundations of my work Interaction modality Multimodality Combining the real and virtual worlds
Combining Combining the real and virtual worlds the real and virtual worlds
Profusion of terms Virtual reality Bit / Atom Non-tangible world Real world Computer Augmented Environment Computer world Virtual world Augmented Video Augmented Interaction Augmented Virtuality Augmented Reality …
Combining Augmented Reality the real and virtual worlds Augmented Virtuality
Common objective Virtual world Real world Augmented Reality VR
Purpose of the task = real world Non-tangible world Real world Computer world Virtual world Virtual world Real world Augmented Virtuality VR
Purpose of the task = computer tangible UI gestural UI graphical UI keyboard UI embodied UI
Laurence Nigay - [email protected] 18 JIRC - Orléans Mercredi 29 Mai 2013
Augmented Reality Augmented Reality Augmented Virtuality Augmented Virtuality
Au gm e nt ed Purpose of the task
ex ec ut io n Real Go a l an d Augmented Reality Real world (AR) world or Virtual world (AV) tas k Augmented Augmented Execution Evaluation Virtual Augmented Virtuality world Au gm e nt ed Type of
ev al ua tio n Execution Evaluation augmentation
Augmented Reality Augmented Reality Augmented Virtuality Augmented Virtuality
Interaction modality Interaction modality
Purpose of the task Purpose of the task
Real Augmented Reality Real Augmented Reality world world
Virtual Virtual Augmented Virtuality Augmented Virtuality world world Type of Type of Execution Evaluation augmentation Execution Evaluation augmentation (Input modality) (Output modality) (Input modality) (Output modality)
Augmented Virtuality Augmented Virtuality
New input modality on mobile devices Head tracking intuitive improves both input and output capabilities Inspired by natural movements Input: Head movements to navigate Output: 3D effect
Virtual world
based on head tracking Execution Evaluation with the front facing camera (Input modality) (Output modality)
Laurence Nigay - [email protected] 19 JIRC - Orléans Mercredi 29 Mai 2013
Augmented Virtuality Augmented Virtuality
Head-coupled perspective for 3D display Off-axis projection
Display Display Display
View looking View looking View looking Forward Left Left Standard display Head-coupled display
Augmented Virtuality Augmented Virtuality
Off-axis projection Well adapted for 3D UI
Augmented Virtuality Outline
Video: viewed more than 2 million times – various Research framework articles (Wired 2011, MacStories 2011, …) Scientific themes Application on AppStore: Research approach i3D: Glasses-free monocular 3D Ubiquitous computing Downloaded 1,5 million times Three research axes Foundations of my work Interaction modality Multimodality Combining the real and virtual worlds Conclusion
Laurence Nigay - [email protected] 20 JIRC - Orléans Mercredi 29 Mai 2013
In the 80’s, Brian Gaines introduced a model on how science technology develops Conclusion over time
Modality and multimodality: A VAST space of possibilities to be explored
Augmented Virtuality / Augmented Reality: Multimodality: Real world (Action/Perception) Digital world (Action/Perception)
Multimodality is an integrating vector for several recent interaction paradigms that include: augmented reality 1980: augmented virtuality Today: Richard Development Today: tangible interfaces Bolt tools for Interaction MIT replication modelling
Thank you ?
Laurence Nigay - [email protected] 21