Mobile Interaction with the Real World
Total Page:16
File Type:pdf, Size:1020Kb
Mobile Interaction with the Real World Dr. Enrico Rukzio Lecturer in Mobile HCI Computing Department Lancaster University (UK) Outline • Mobile interaction with the real world – Overview, related research areas, classification • Focus on – Touching (NFC based interaction techniques) – Pointing (personal projectors) – Scanning and user mediated object selection • Application areas • Summary & Conclusion 2 Mobile Interaction with the Real World sensors in mobile device interaction with augmented and not augmented “things” Exchange (images, audio files, messages), to {call, text, play, Play (mobile gaming) surf the internet}, PIM proximity of users Location based services (e.g. tour guide, city guide, mobile navigation) [Kindberg et al. 2002, Rukzio 2007] Related Research Areas R R R C C Before the computer Usage of everyday computers Virtual Reality (laptop, mobile phone) R R R C C C C C C Ubiquitous Computing Mobile Interaction Mobile Augmented Reality (Real World Computer) with the Real World C – Computer Human – Computer - Interaction R – Real World Human – Real World - Interaction Real World – Computer - Interaction [Rukzio 2007] based on Computer – Computer - Interaction [Rekimoto and Nagao 1995] Mobile Interaction with the Real World: Interaction Techniques Interaction Touching Pointing Scanning User-mediated Indirect Remote technique object interaction Controls Description The user touches The user points A link between The user types in The user controls a smart object on a smart mobile device and information a remote display with a mobile object with a smart object is provided by the with a mobile device to mobile device to established object to establish device. establish a link. establish a link. because of their a link between proximity. them. Illustration [Välkkynen et al. 2003] Real world Distance object - Distance object - Maximal distance Line of sight and Line of sight, aspects mobile device: mobile device: object - mobile readable often indoors 0...10 cm, line of 10 cm...10 m, device: 100m sight line of sight Device - Radio: RFID, Visual: visual Location: No direct link Communication: Smart Object NFC, proximity marker, light Bluetooth, WLAN, Bluetooth, GPRS, Interaction sensors beam, IrDA GPS UMTS [Rukzio 2007] Touching: Introduction • Select and interact with object by touching it with the mobile device – Intuitive, direct interaction, user has to be nearby, augmentation • [Want et al. 1999] (Xerox PARC) – One of the first who presented a prototype: RFID tags + RFID reader connected to a mobile device (tablet computer) – Applications: augmented books, documents and business cards RFID tags provide links to corresponding services ordering a book / picking up an email address • [Välkkynen et al. 2003] (VTT) – Further implementation called TouchMe – Based on proximity sensors (IR) which sense the distance between augmented object and mobile device Touching: NFC / RFID • Products & prototypes based on short range passive RFID and Near Field Communication (NFC) [Want 2006] • NFC (Near Field Communication) – Short range data communication technology (13.56 MHz) – Standardized: Near Field Communication - Interface and Protocol NFCIP 1/2 (ECMA-340, ECMA-352, ISO/IEC 18092) – Compatible to MIFARE (ISO/IEC 14443A), FeliCa (complies with ISO/IEC 18092) and ISO/IEC 15693. – Several NFC phones available (e.g. Nokia 6212/6131 NFC) – Future • Håkan Djuphammar (Ericsson's VP of systems architecture): “A year from now, basically every new phone that's sold will have [Near Field Communication]” [Sherwood 2009] • Costs to add NFC to phones $1 by 2012/2013 [Clark 2009] • http://www.nfc-forum.org/news/ 7 Touching: NFC interaction styles NFC Device Tag NFC Chip Information RFID NFC Device Interaction Point Contactless NFC Chip Reader smart card NFC Device NFC Device NFC Chip NFC Chip Bluetooth [Rukzio 2007] Touch based mobile applications • Tag: How many tags are on the object? • Single: Link to corresponding service • Multi: Link to many services / options • Display: Which kind of object is touched? • Static: e.g. poster or newspaper, no feedback provided by display • Multi: e.g. projection or LCD display, feedback by display and mobile phone Tag / Display Static Dynamic Single Interact with sensors, Interaction with laptops, payment and ticketing access control, payment readers readers Multi Marked-up maps, Touch & Touch & interact Interact 9 Single tag / static display [missphones 2009] [Touch & Travel] [VTT 2007] [missphones 2009] 10 Single tag: NTT DOCOMO FeliCa, Osaifu-Keitai & ToruCa • i-mode FeliCa allows a mobile phone to perform the function of traditional Sony FeliCa cards with additional functionality • Osaifu-Keitai are ‘mobile wallets’ that use the system • ToruCa: coupon service • Restaurant flyers • Promotional coupons • Compatible handsets in Japan (2009) • DoCoMo: 29 million • Softbank: 10 million • KDDI: >10 million • Mobile wallet supported by 640.000 stores [i-mode FeliCa] Single tag / dynamic display [ECMA 2004] [Dhiram 2007] [Dhiram 2007] 12 Multi tag / static display: Marked-up Maps • Selection – M locations – 1 out of M – N out of M [Reilly et al . 2006] Multi tag / static display: Touch & Interact [Hardy and Rukzio 2008] Multitag / static display: Meal service for elderly people • Selection – M meals – 1 out of M [Häikiö et al. 2007] Multitag / static display: RFID McDonalds • RFID dongle augments a standard Korean mobile phones with a standard hardware interface • ‘Touch order’ is a system which allows selection and payment to be made from a menu at every table • Secure NFC phone applications are used for billing (phone bill) and data services • RFID cards are trialed in the US for use at the drive-thru. • Selection, guidance ? [Nearfield 2007] 16 Multitag / static display: cinema poster • Tasks – Order movie ticket – View movie details • Selections – C cinemas, P persons, T timeslot – [1, X] out of C – [Y] out of P – [Z] out of T • Guidance: visual on the poster, on mobile phone [Broll et al. 2007] Multitag / dynamic display Touch & Interact • User interface: mobile phone & dynamic display [Hardy and Rukzio 2008] Multitag / dynamic display Touch & Interact [Hardy and Rukzio 2008] Multitag / dynamic display Touch & Interact Coordinates assigned to each tag in the mesh [Hardy and Rukzio 2008] Multitag / dynamic display Touch & Interact • Simple upload and download of pictures from mobile phone to laptop upload download [Seewoonauth et al. 2009] 21 NFC: granularity issue • Problem: NFC tag size & phone size [Hardy and Rukzio 2008] Pick & Drop [Rekimoto 1997] Touch based mobile interaction with interactive surfaces Visual marker (Byte and Identity Tags) [Carpenter 2008, surface.com] BlueTable (Shape detection & IRDA blinking) 1. Detect the placement of a new phone-shaped object by visual means. 2. For each switched-on Bluetooth device: • a. Attempt to connect to the device over Bluetooth. Continue if the device advertises globally unique identifier (GUID), else move on to the next device. • b. Command the device to blink its IRDA (infrared) port. • c. If the blink is detected at the position of the object go to step 3, else move on to the next Bluetooth device. 3. Determine the exact orientation of the device (optional). [Wilson & Sarin 2007] 24 Pointing: Introduction • Select or control an object by pointing on it with the mobile device • Intuitive, direct interaction, user has to be nearby, often augmentation • [Fitzmaurice 1993] (University of Toronto) • Using mobile devices for pointing based interactions to interact with related services • Application: map on which the user can point to get additional information / a computer augmented library • [Rekimoto, Nagao 1995] (Sony Computer Science Laboratory) • NaviCam project / prototype • Visual markers interpreted by a camera attached to the mobile device • Markers: 4-bit visual colour codes (sequence of red and blue stripes), size of 3cm x 5cm • Distance between device and object: 30-50 cm 25 Pointing: Implementations Principle Visual Marker Image Light beam Personal Infrared Recognizing the Recognition projector mobile device Illustration References [Rekimoto and [Föckler et al. [Välkkynen et [Raskar et al. [Ailisto et al. [BahnHandyTick Nagao 1995] 2005] al. 2003] 2004] 2003] et, Miyaoku et al. 2004] Advantages Makers are Smart objects Very natural Project user IrDA is Handy simple, do not need to interaction interface onto integrated in possibility for inexpensive, be augmented. (remote control) object many mobile identification disposable. No No power devices. (tickets, etc.) power supply. supply. Disadvan- Visual Great demands Smart object Tracking Smart object Code (e.g. paid tages obtrusiveness, on image must provide a must be ticket) can not limited storage recognition and feedback enhanced by be shown when capabilities data model. channel (RF, IrDA phone is out of Bluetooth, etc.) functionalities. power. [Rukzio 2007] Pointing: Implementations: Visual Marker • Uses the built-in camera of mobile phones, different marker types • NTT DoCoMo / 2003 – first mobile phones (505i Series) with a preinstalled QR code application, 30 million mobile phones with QR code suppport [Fowler 2005] • Two-dimensional codes – Can store more information then one- dimensional codes (EAN-13 bar codes) – Comprehensive overview NaviCam QR Codes Visual Codes in [Rohs 2005] [Rekimoto, [QRCode] [Rohs and Nagao 1995] Gfeller 2004] • Application areas: – Advertisement posters, magazines, newspapers 27 Pointing: Implementations: Image recognition • [Fritz et al. 2004]