Post-Wimp Interaction With Desktop Computers New Tools and Techniques Pierre Dragicevic 1 WIMP Interaction n WIMP = Windows, Icons, Menus, Pointer + n Standard Devices: n Standard Interaction Techniques: n Mouse n Windows, widgets, icon n Keyboard n Text input, click, double-click, d&d 2 What Next and Why? n WIMP system (Xerox Star, 1979) 3 Post-WIMP Interaction Paradigms Better exploit our motor abilities: n Bimanual Interaction n « Multifinger » Interaction Map Zooming [Hinckley et al 98] 4 SmartSkin [Rekimoto 2002] Post-WIMP Interaction Paradigms Better exploit our perceptual abilities: n Information visualization n Enhanced displays TreeMaps [Schneiderman et al] Multiple screens & large displays 5 Post-WIMP Interaction Paradigms Make interaction more « direct »: n Gestural interaction n See-through tools Toolglasses [Bier et al 93] 6 Two Research Directions n Explore Post-WIMP interaction further: n Propose innovative solutions to known problems n Investigate alternative interaction paradigms n Disseminate existing research: n Integrate Post-WIMP paradigms into GUI toolkits n Provide new design guidelines 7 My Research n Explore Post-WIMP interaction further: n Propose innovative solutions to known problems n Investigate new interaction paradigms n Disseminate existing research: n Integrate Post-WIMP paradigms into GUI toolkits n Provide new design guidelines n Support the exploration of new approaches: n Study more usable and more flexible tools n Advocate a vision of Post-WIMP with sound examples 8 My Research New tools New techniques 9 Introduction I. Input Configurator New tools II. Artistic Resizing III. SpiraClock New techniques IV. Fold n’ Drop Conclusion 10 Introduction I. Input Configurator I. Input Configurator II. Artistic Resizing An Input Interaction Model for III. SpiraClock Highly Configurable Multi-Device Interactive Systems IV. Fold n’ Drop Pierre Dragicevic Jean-Daniel Fekete Conclusion 11 Goal: Support Alternative Input n WIMP system n Input-Open System (Xerox Star, 1979) 12 Why Using Alternative Input? n Augmenting the bandwith n Optimizing interaction according to: the task the user the environnement 13 Current State of the Art n Very poor support for alternative input devices in: n Current operating systems n Current applications n Current toolkits n Little research on input management n Post-WIMP tools are specialized towards: n A fixed set of input devices n A fixed set of interaction techniques n No approach deals with: n Multiple and heterogeneous input (enriched / impoverished) n Flexibility and configurability 14 The Input Configuration Model n Approach: n Fully separate physical devices from applications n Provide a flexible way to map devices to applications 15 The Input Configuration Model n Device: n Parametrizable black box n Processes input into output n I/O Slots: Input Output n Simple types slots slots n + compound types n Implicit I/O: n User input Implicit Implicit input output n Feedback 16 The Input Configuration Model n Device types: n System (hardware ressources) Input Configuration n Adapter (software processes) n Application (specific to each app) n Input Configuration: n A set of system and application devices connected through adapter devices System Adapter Application 17 devices devices devices The ICON Toolkit n System Devices: 18 The ICON Toolkit n Adapter devices: n Data processing 19 The ICON Toolkit n Adapter devices: n Data processing n Graphical feedback 20 The ICON Toolkit n Adapter devices: n Data processing n Graphical feedback n Interaction techniques 21 The ICON Toolkit n Application Devices: n Example: IConDraw tools 22 The ICON Toolkit n Graphical Toolkit Devices: n Example: Swing devices 23 The ICON Toolkit n The input configuration editor: EditingEditing panepane AvailableAvailable devicesdevices 24 The ICON Toolkit n The input configuration editor 25 The ICON Toolkit n The input configuration editor n Dynamic wire drawing algorithm 26 The ICON Toolkit n The input configuration editor n Fluid expand/collapse 27 The ICON Toolkit n The input configuration editor n Fisheye-drag 28 Examples 1. Exploiting rich input n Pressure-sensitive drawing inside IConDraw 29 Examples 1. Exploiting rich input 30 Examples 1. Exploiting rich input n Symmetric bimanual interaction in IConDraw 31 Examples 1. Exploiting rich input 32 Examples 1. Exploiting rich input n Integral control of the Responsive Face [Perlin 97] 33 Examples 1. Exploiting rich input n Integral control of the Responsive Face [Perlin 97] 34 Examples 1. Exploiting rich input 35 Examples 2. Improving Accessibility n Keyboard ? Pointing 2nd order control 36 Examples 2. Improving Accessibility n Pointer ? Text editing « Floating QuikWriting » [Perlin] 37 Examples 2. Improving Accessibility n Pointer ? Text editing « Floating QuikWriting » [Perlin] 38 Examples 3. Augmenting WIMP applications n « Augmented pointing » Filter Events 39 Examples 3. Augmenting WIMP applications n « Augmented pointing » : smoothed cursor 40 Examples 3. Augmenting WIMP applications n « Augmented pointing » : throwable cursor 41 Examples 3. Augmenting WIMP applications n « Augmented pointing » : speech cursor rightNoiseNoise…… right releasereleasedowndown slowerslower presspressstopstopupup 42 Examples 3. Augmenting WIMP applications n « Generalized Drag and Drop » 43 Projects using ICON n Marina II (2001) Photograph-based architectural modelling tool n Svalabard [Huot et al 2003] Post-WIMP sketching-based 3D modelling interface n MaggLite [Huot et al 2004] Post-WIMP UIMS based on ICON n LRI, University of Orsay [Appert et al 2003] Experimental comparison of WIMP / Post-WIMP techniques n Smart Home and Impairment (INT/ENST/EDF 2005) Configurable software architecture for accessibility n Student projects 44 Introduction I. Input Configurator II. Artistic Resizing II. Artistic Resizing A Technique for Rich III. SpiraClock Scale-Sensitive Vector Graphics Pierre Dragicevic IV. Fold n’ Drop Stéphane Chatty David Thevenin Jean-Luc Vinot Conclusion 45 Graphically Rich User Interfacs n Graphic designers are given more and more importance in GUIs n Exploitation of vector graphics (e.g, Scalable Vector Graphics) n IntuiKit [Chatty et al 2004] 46 IntuiKit: Example of Team Work Programming Low-fi prototyping Integration Graphic design 47 The Resizing Problem n Fixed size n Naive scaling n Common resizing 48 Expressing Resizing n Resizing is commonly described using formulae w xL · x = (w-w ) / 2 · w = 5 r L L B yL · yL = (h-hL) / 2 · hB = 5 h hL · wL = 20 · hL = 10 · r = 20 wL hB wB n These formulae are: n Translated into code by the programmer n Or used as an input to constraint-solving systems 49 Expressing Resizing n But designers think visually n They are used to produce visual variants On white On black On red Medium Small paper paper paper size size n Variants are also useful to convey general laws 50 Artistic Resizing 1. Designers produce variants 2. IntuiKit interprets using their authoring tool the example set 51 Artistic Resizing A scenario 52 Artistic Resizing A scenario 53 Artistic Resizing A scenario 54 Artistic Resizing A scenario 55 Artistic Resizing A scenario 56 Artistic Resizing How does it work? n Based on local interpolation of transformations T1 T1’ 57 Artistic Resizing How does it work? n Each variant of T1 is associated with the example’s bounding box T1 T1’ ? T1’’ T1’’’ 58 Artistic Resizing Orthogonal Interpolation n Orthogonal Interpolation hypothesis: n Width only impacts the X contribution n Height only impacts the Y contribution 11 a12 a13 width X a X0 21 a22 a23 height Y = a Y0 1 0 0 1 1 59 Artistic Resizing Orthogonal Interpolation n Piecewise linear interpolations are performed along width and height separately: height a23 a22 a21 a11a12 a13 width 60 Artistic Resizing Properties of Orthogonal Interpolation n Preserves algebraic measures: n If horizontal or vertical distances are the same on the examples they will remain the same 61 Artistic Resizing Properties of Orthogonal Interpolation n Preserves relative ratios: n Ratios that are invariant on the examples will remain invariant 62 Artistic Resizing Properties of Orthogonal Interpolation n Preserves contacts: n Two points that coincide on the examples will always coincide 63 Artistic Resizing Properties of Orthogonal Interpolation n Preserves parallelism: n Two lines that are parallel on the examples will remain parallel 64 Artistic Resizing n Supports invariants but also finely tuned behaviors w xL r yL h hL wL hB wB · xL = (w-wL) / 2 · wB = 5 · yL = (h-hL) / 2 · hB = 5 · wL = 20 · hL = 10 · r = 20 65 Artistic Resizing Blending with traditional layout managers n Containment hierarchies: 66 Artistic Resizing Blending with traditional layout managers n Inside an « aqua-dock » layout manager: 67 Artistic Resizing vs. Constraint Inference n Demonstrational layout specification systems: n Peridot [Myers 90] n Chimera [Kurlander & Feiner 93] n Constraint inference: n Orthogonal interpolation: n Extensive search n Ensures the preservation of for invariants « interesting » invariants n Sensitive to combi- n Fast computation, even with natorial explosion complex vector graphics n Over-constrained or n Limitations are known, unintended rules results are easy to predict 68 Introduction III. SpiraClock I. Input Configurator II. Artistic Resizing A Continuous and Non-Intrusive III. SpiraClock Technique for Upcoming Events IV. Fold n’ Drop Pierre Dragicevic Stéphane Huot Conclusion 69 Time Management Tools n Calendars displays: n Good for long-term time n Reminders and management