ubiquitous computing and augmented realities

chapter 20 • ubiquitous computing – filling the real world with computers ubiquitous computing • virtual and augmented reality and augmented realities – making the real world in a computer!

Challenging HCI Assumptions Ubiquitous Computing

• What do we imagine when we think of a • Any computing technology that permits computer? human interaction away from a single “The most profound technologies workstation are those that disappear.” • Implications for Weiser – Technology defining the interactive experience • 1990’s: this was not our imagined – Applications or uses computer! – Underlying theories of interaction

Scales of devices Device scales

• Weiser proposed •Inch –Inch –PDAs –Foot – PARCTAB –Yard – Voice Recorders –smart phones • Implications for device size as well as • Individuals own many relationship to people of them and they can all communicate with each other and environment.

1 Device scales Device scales

• Foot •Yard – notebooks – electronic whiteboards –tablets – plasma displays – digital paper – smart bulletin boards • Individual owns • Buildings or several but not institutions own them assumed to be and lots of people always with them. share them.

Defining the Interaction Different Inputs Experience • Implicit input – Sensor-based input – Extends traditional explicit input (e.g., keyboard and mouse) – Towards “awareness” – Use of recognition technologies – Introduces ambiguity because recognizers Capacitive sensing on a table Sensors on a PDA are not perfect

Multi-scale and distributed The output experience output • Screens of many sizes • More than eye-grabbing raster displays – (very) small – Ambient: use features of the physical environment to signal information – (very) large – Peripheral: designed to be in the background •Examples: – The Dangling String – The Water Lamp (shown) • Distributed in space, but coordinated

2 Merging Physical and Digital Application Themes Worlds

•How can we remove A “digital” desk •Context-aware computing the barrier? – Sensed phenomena facilitate easier – Actions on physical interaction objects have meaning • Automated capture and access electronically, and – Live experiences stored for future access vice versa • Toward continuous interaction –Output from electronic world – Everyday activities have no clear begin-end superimposed on conditions physical world An augmented calendar

New Opportunities for Theory Evaluation Challenges

• Knowledge in the world • How can we adapt other HCI techiques – Ubicomp places more emphasis on the physical world to apply to ubicomp settings? • Activity theory – Goals and actions fluidly adjust to physical state of – Ubicomp activities not so task-centric world – Technologies are so new, it is often hard to • Situated action and distributed cognition get long-term authentic summative – Emphasizes improvisational/opportunistic behavior evaluation versus planned actions – Metric of success could be very different • Ethnography (playfulness, non-distraction versus – Deep descriptive understanding of activities in context efficiency)

ambient wood City - shared experience

• real wood! … filled with electronics • visitors to Mackintosh Interpretation Centre • light and moisture meters – some on web, some use VR, some really there – recorded with GPRS location • interacting –drawn on map later – talk via microphones • ‘periscope’ – ‘see’ each other virtually – shows invisible things – uses RFID • different places • triggered sound • different modalities • shared experience

3 virtual and augmented reality

VR - technology & experience web, desktop and simulators AR – mixing virtual and real

virtual reality technology VR headsets

• headsets allow user to “see” the virtual world • small TV screen for each eye • gesture recognition achieved with DataGlove • slightly different angles (lycra glove with optical sensors that measure hand and finger positions) • 3D effect • eyegaze allows users to indicate direction with eyes alone • whole body position sensed, walking etc.

immersion on the desktop

•VR • headset VR – computer simulation of the real world – expensive, uncomfortbale • mainly visual, but sound, haptic, gesture too • desktop VR – experience life-like situations – use ordinary monitor and PC • too dangerous, too expensive • cheap and convenient – see unseen things: •in games … • too small, too large, hidden, invisible – e.g. manipulating molecules •and on the web – VRML – virtual reality markup language • the experience – aim is immersion, engagement, interaction

4 VRML … VR on the web command and control

#VRML V1.0 ascii • scenes projected on walls Separator { Separator { # for sphere • realistic environment Material { • hydraulic rams! emmissiveColor 0 0 1 # blue } • real controls Sphere { radius 1 } } •other people Transform { translation 4 2 0 } Separator { # for cone Texture2 { filename "big_alan.jpg" •for: } – flight simulators Cone { radius 1 # N.B. width=2*radius –ships height 3 – military } } }

augmented reality (AR) applications of AR

• images projected over the real world maintenance – aircraft head-up display – overlay instructions – semi-transparent goggles – display schematics – projecting onto a desktop • types of information – unrelated – e.g. reading email with wearable examples – related – e.g. virtual objects interacting with world – photocopier engineers • issues • registration critical arrows point to parts – registration – aligning virtual and real – aircraft wiring looms – eye gaze direction • registration perhaps too hard, use schematic

applications of VR

• simulation – games, military, training • VR holidays information and data visualisation – rainforest, safari, surf, ski and moon walk … all from your own armchair •medical –surgery VR, 3D and 2D displays • scans and x-rays used to build model scientific and complex data then ‘practice’ operation • force feedback best interactivity central – phobia treatment • virtual lifts, spiders, etc.

5 scientific and technical data virtual wind tunnel

• number of virtual dimensions that are ‘real’ • fluid dynamics to simulate air flow • three dimensional space • virtual bubbles used to show movements – visualise invisible fields or values – e.g. virtual wind tunnel • ‘better’ than real • two dimensional space wind tunnel … – can project data value up from plane – no disruption of – e.g. geographic data air flow – N.B. viewing angle hard for static visualisation – cheaper and faster • no ‘real’ dimensions – 2D/3D histograms, scatter plots, pie charts, etc.

structured informnation visualising hiererchy

• scientific data – just numbers • 2D organisation chart • information systems … lots of kinds of data – familiar representation – what happens when it gets wide? • hierarchies managing – file trees, organisation charts director •networks – program flow charts, hypertext structure sales marketing production • free text … manager manager manager – documents, web pages

F. Bloggs F. Bloggs A. Jones R.Carter K. West B. Firth

J. Smith P. Larkin

wide hierarchies … use 3D? networks in 2D

managing director • network or ‘graph’: – nodes – e.g. web pages sales marketing production manager manager manager – links – may be directed or not – e.g. links • planar – can drawn without crossing F. Bloggs F. Bloggs A. Jones R.Carter K. West B. Firth J. Smith P. Larkin •non-planar – any 2D layout has crossings

• cone trees (Xerox) • levels become rings • overlap ‘OK’ in 3D

Planar graph Non-planar graph

6 time and interactivity

• visualising in time – time dimension mapped to space – changing values: sales graphs, distance-time – events: Gantt chart, timelines, historical charts e.g. Lifelines – visualising medical and court records •using time – data dimension mapped to time – time to itself: fast/slow replay of events – space to time: Visible Human Project • interactivity – change under user control e.g. influence explorer

between two worlds

• ubiquitous computing – computers fill the real world • virtual reality and visualisation – real world represented in the computer • augmented reality, ambient displays … – physical and digital intermingled

…maturity – VR and visualisation – commonplace – AR, ubiquity … coming fast!

7