Post-Wimp Interaction With Desktop Computers
New Tools and Techniques
Pierre Dragicevic
1 WIMP Interaction
n WIMP = Windows, Icons, Menus, Pointer
+
n Standard Devices: n Standard Interaction Techniques:
n Mouse n Windows, widgets, icon n Keyboard n Text input, click, double-click, d&d 2 What Next and Why?
n WIMP system (Xerox Star, 1979)
3 Post-WIMP Interaction Paradigms
Better exploit our motor abilities:
n Bimanual Interaction
n « Multifinger » Interaction
Map Zooming [Hinckley et al 98]
4 SmartSkin [Rekimoto 2002] Post-WIMP Interaction Paradigms
Better exploit our perceptual abilities:
n Information visualization
n Enhanced displays
TreeMaps [Schneiderman et al]
Multiple screens & large displays 5 Post-WIMP Interaction Paradigms
Make interaction more « direct »:
n Gestural interaction
n See-through tools
Toolglasses [Bier et al 93] 6 Two Research Directions n Explore Post-WIMP interaction further:
n Propose innovative solutions to known problems n Investigate alternative interaction paradigms
n Disseminate existing research:
n Integrate Post-WIMP paradigms into GUI toolkits n Provide new design guidelines
7 My Research n Explore Post-WIMP interaction further:
n Propose innovative solutions to known problems n Investigate new interaction paradigms
n Disseminate existing research:
n Integrate Post-WIMP paradigms into GUI toolkits n Provide new design guidelines n Support the exploration of new approaches:
n Study more usable and more flexible tools n Advocate a vision of Post-WIMP with sound examples 8 My Research
New tools
New techniques
9 Introduction
I. Input Configurator
New tools II. Artistic Resizing
III. SpiraClock
New techniques IV. Fold n’ Drop
Conclusion 10 Introduction
I. Input Configurator I. Input Configurator
II. Artistic Resizing
An Input Interaction Model for III. SpiraClock Highly Configurable Multi-Device Interactive Systems
IV. Fold n’ Drop
Pierre Dragicevic Jean-Daniel Fekete Conclusion 11 Goal: Support Alternative Input
n WIMP system n Input-Open System (Xerox Star, 1979)
12 Why Using Alternative Input? n Augmenting the bandwith n Optimizing interaction according to:
the task the user the environnement 13 Current State of the Art n Very poor support for alternative input devices in:
n Current operating systems n Current applications n Current toolkits n Little research on input management n Post-WIMP tools are specialized towards:
n A fixed set of input devices n A fixed set of interaction techniques n No approach deals with:
n Multiple and heterogeneous input (enriched / impoverished)
n Flexibility and configurability 14 The Input Configuration Model n Approach:
n Fully separate physical devices from applications
n Provide a flexible way to map devices to applications
15 The Input Configuration Model
n Device:
n Parametrizable black box n Processes input into output
n I/O Slots: Input Output n Simple types slots slots n + compound types
n Implicit I/O:
n User input Implicit Implicit input output n Feedback
16 The Input Configuration Model n Device types:
n System (hardware ressources) Input Configuration n Adapter (software processes) n Application (specific to each app) n Input Configuration:
n A set of system and application devices connected through adapter devices
System Adapter Application 17 devices devices devices The ICON Toolkit n System Devices:
18 The ICON Toolkit
n Adapter devices:
n Data processing
19 The ICON Toolkit
n Adapter devices:
n Data processing n Graphical feedback
20 The ICON Toolkit
n Adapter devices:
n Data processing n Graphical feedback n Interaction techniques
21 The ICON Toolkit
n Application Devices:
n Example: IConDraw tools
22 The ICON Toolkit
n Graphical Toolkit Devices:
n Example: Swing devices
23 The ICON Toolkit
n The input configuration editor:
EditingEditing panepane
AvailableAvailable devicesdevices
24 The ICON Toolkit
n The input configuration editor
25 The ICON Toolkit n The input configuration editor
n Dynamic wire drawing algorithm
26 The ICON Toolkit n The input configuration editor
n Fluid expand/collapse
27 The ICON Toolkit n The input configuration editor
n Fisheye-drag
28 Examples
1. Exploiting rich input
n Pressure-sensitive drawing inside IConDraw
29 Examples
1. Exploiting rich input
30 Examples
1. Exploiting rich input
n Symmetric bimanual interaction in IConDraw
31 Examples
1. Exploiting rich input
32 Examples
1. Exploiting rich input
n Integral control of the Responsive Face [Perlin 97]
33 Examples
1. Exploiting rich input
n Integral control of the Responsive Face [Perlin 97]
34 Examples
1. Exploiting rich input
35 Examples
2. Improving Accessibility
n Keyboard ? Pointing
2nd order control 36 Examples
2. Improving Accessibility
n Pointer ? Text editing
« Floating QuikWriting » [Perlin] 37 Examples
2. Improving Accessibility
n Pointer ? Text editing
« Floating QuikWriting » [Perlin] 38 Examples
3. Augmenting WIMP applications
n « Augmented pointing »
Filter Events
39 Examples
3. Augmenting WIMP applications
n « Augmented pointing » : smoothed cursor
40 Examples
3. Augmenting WIMP applications
n « Augmented pointing » : throwable cursor
41 Examples
3. Augmenting WIMP applications
n « Augmented pointing » : speech cursor
rightNoiseNoise…… right releasereleasedowndown slowerslower presspressstopstopupup
42 Examples
3. Augmenting WIMP applications
n « Generalized Drag and Drop »
43 Projects using ICON
n Marina II (2001) Photograph-based architectural modelling tool
n Svalabard [Huot et al 2003] Post-WIMP sketching-based 3D modelling interface
n MaggLite [Huot et al 2004] Post-WIMP UIMS based on ICON
n LRI, University of Orsay [Appert et al 2003] Experimental comparison of WIMP / Post-WIMP techniques
n Smart Home and Impairment (INT/ENST/EDF 2005) Configurable software architecture for accessibility
n Student projects 44 Introduction
I. Input Configurator II. Artistic Resizing
II. Artistic Resizing
A Technique for Rich
III. SpiraClock Scale-Sensitive Vector Graphics
Pierre Dragicevic IV. Fold n’ Drop Stéphane Chatty David Thevenin Jean-Luc Vinot Conclusion 45 Graphically Rich User Interfacs
n Graphic designers are given more and more importance in GUIs
n Exploitation of vector graphics (e.g, Scalable Vector Graphics)
n IntuiKit [Chatty et al 2004]
46 IntuiKit: Example of Team Work
Programming
Low-fi prototyping Integration
Graphic design
47 The Resizing Problem
n Fixed size
n Naive scaling
n Common resizing
48 Expressing Resizing n Resizing is commonly described using formulae
w
xL · x = (w-w ) / 2 · w = 5 r L L B yL · yL = (h-hL) / 2 · hB = 5 h hL · wL = 20 · hL = 10 · r = 20 wL hB
wB n These formulae are:
n Translated into code by the programmer n Or used as an input to constraint-solving systems
49 Expressing Resizing n But designers think visually n They are used to produce visual variants
On white On black On red Medium Small paper paper paper size size
n Variants are also useful to convey general laws
50 Artistic Resizing
1. Designers produce variants 2. IntuiKit interprets using their authoring tool the example set
51 Artistic Resizing A scenario
52 Artistic Resizing A scenario
53 Artistic Resizing A scenario
54 Artistic Resizing A scenario
55 Artistic Resizing A scenario
56 Artistic Resizing How does it work? n Based on local interpolation of transformations
T1 T1’
57 Artistic Resizing How does it work? n Each variant of T1 is associated with the example’s bounding box
T1 T1’ ?
T1’’ T1’’’
58 Artistic Resizing Orthogonal Interpolation n Orthogonal Interpolation hypothesis:
n Width only impacts the X contribution n Height only impacts the Y contribution
11 a12 a13 width X a X0
21 a22 a23 height Y = a Y0
1 0 0 1 1
59 Artistic Resizing Orthogonal Interpolation n Piecewise linear interpolations are performed along width and height separately:
height a23 a22 a21
a11a12 a13
width
60 Artistic Resizing Properties of Orthogonal Interpolation
n Preserves algebraic measures:
n If horizontal or vertical distances are the same on the examples they will remain the same
61 Artistic Resizing Properties of Orthogonal Interpolation
n Preserves relative ratios:
n Ratios that are invariant on the examples will remain invariant
62 Artistic Resizing Properties of Orthogonal Interpolation
n Preserves contacts:
n Two points that coincide on the examples will always coincide
63 Artistic Resizing Properties of Orthogonal Interpolation
n Preserves parallelism:
n Two lines that are parallel on the examples will remain parallel
64 Artistic Resizing
n Supports invariants but also finely tuned behaviors
w
xL r yL h hL
wL hB
wB · xL = (w-wL) / 2 · wB = 5 · yL = (h-hL) / 2 · hB = 5 · wL = 20 · hL = 10 · r = 20 65 Artistic Resizing Blending with traditional layout managers
n Containment hierarchies:
66 Artistic Resizing Blending with traditional layout managers
n Inside an « aqua-dock » layout manager:
67 Artistic Resizing vs. Constraint Inference n Demonstrational layout specification systems:
n Peridot [Myers 90] n Chimera [Kurlander & Feiner 93]
n Constraint inference: n Orthogonal interpolation:
n Extensive search n Ensures the preservation of for invariants « interesting » invariants n Sensitive to combi- n Fast computation, even with natorial explosion complex vector graphics n Over-constrained or n Limitations are known, unintended rules results are easy to predict 68 Introduction III. SpiraClock I. Input Configurator
II. Artistic Resizing
A Continuous and Non-Intrusive III. SpiraClock Technique for Upcoming Events
IV. Fold n’ Drop Pierre Dragicevic Stéphane Huot
Conclusion 69 Time Management Tools
n Calendars displays:
n Good for long-term time n Reminders and management Alarms:
n But useless for n Prevent forgetting upcoming events n But too intrusive n Don’t allow continuous anticipation
70 SpiraClock
n Analog desktop clock
n Events are displayed as sectors
71 SpiraClock
n Event are aligned with minute marks
n Relative times can be read directly
72 SpiraClock
n One example: viewing bus departures
It’s 18h50. What buses can I catch ?
73 SpiraClock
n One example: viewing bus departures
74 SpiraClock
n One example: viewing bus departures
75 Uses of SpiraClock n BusClock used in Ecole des Mines de Nantes Since 2002
n Other applications:
n ShuttleClock - shuttles of University of Maryland n OutClock - synchronized with MS Outlook n CalendarClock - uses VCal exchange format 76 Introduction IV. Fold n’ Drop I. Input Configurator
II. Artistic Resizing
Combining Paper-Based and III. SpiraClock Crossing-Based Interaction Paradigms for Dragging and Dropping between Overlapping Windows IV. Fold n’ Drop
Pierre Dragicevic
Conclusion 77 Drag & Drop Scenario n Moving a file into a folder
78 Drag & Drop Scenario n Problem: the target is hidden
79 Drag & Drop Scenario n Solution 1: adjust windows
adjust
drag & drop
80 Drag & Drop Scenario n Solution 2: cut & paste
cut
bring to the front
81 Drag & Drop Scenario n Solution 2: cut & paste
paste
82 Using Window Navigation n Dual representation ® Mapping problems (XP taskbar)
n Activation costs (mouse-over timeout on Exposé)
83 Using Window Navigation n Activation is a superfluous concept for D&D tasks
84 The Fold n’ Drop Technique n Folding a window
n The mouse leaves the window
n The mouse leaves then returns back
85 The Fold n’ Drop Technique n Other gestures
n Leafing through
n Discarding a window
n Unfolding
86 Video
87 Fold n’ Drop builds upon…
n Paper-based window metaphors
n The « Folded » effect [Beaudouin-Lafon 01]
Ametista Toolkit [Roussel 03] AttrActive Windows [Denoue 03] 88 Fold n’ Drop builds upon… n Gestural interfaces
n Natural, « physical » gestures
n Do not need a recognition engine
n Naturally blend with direct manipulation [see e.g., Pook et al 00]
n Little interference with other interactions
89 Fold n’ Drop builds upon… n The crossing-based interaction paradigm [Accot & Zhai 02]
n Fn’D uses double-crossing and n-crossing
n Extends the paradigm:
• Point & Click ® Cross boundaries • Drag ® Push boundaries • Double-Click ® Timed double-crossing
n Timing is made visible:
90 Introduction
I. Input Configurator Conclusion II. Artistic Resizing
III. SpiraClock
IV. Fold n’ Drop
Conclusion 91 Conclusion n Common desktop computing activities can still be improved:
n Management of multiple tasks • Task switching • Data transfer between applications n Management of personal data • Viewing, searching • Organizing n Time management • Long-term • Short-term n But how to design Post-WIMP techniques ? 92 Conclusion Post-WIMP GUI Design as a Creative Activity n The success of a technique is difficult to predict:
n Efficiency + pleasingness ? Importance of « trial and error » (iterative design) n But techniques also require fine tuning:
n Timings, input filtering, graphical effects, color design,… ? Need for closer experimentation-evaluation loops n « Creative tools » allow to explore and test by ourselves:
n Flexible, open n High degree of control, finely tunable n Fast experimentation-evaluation loop n Simple and predictable
93 Conclusion 5 characteristics of Post-WIMP Techniques
1. Specificity 2. High I/O bandwidth
n Multiple input channels n Advanced output devices or rich graphics 3. Directness
n Stimulus/Response compatibility 4. Reactivity and Fluidity
n Responsive and continuous feedback 5. Straightforwardness
n Familiar metaphors n Simple, predictable system behavior 94 Conclusion About Predictability n Example: gestures in Fold n’ Drop
« I missed the fold »
« I returned back too late » 95 Conclusion About Predictability n Another example: freehand polyline drawing
n Using a smart tool (stroke beautifier):
96 Conclusion About Predictability n Another example: freehand polyline drawing
n Using a smart tool (stroke beautifier):
97 Conclusion About Predictability n Another example: freehand polyline drawing
n Using a dumb tool (spring-loaded cursor):
98 Conclusion Future Work n Continue to explore new interaction paradigms
n Desktop n Non-desktop n Generalize and theorize:
n Intelligence vs predictability n Explicit gestures n Augmented WIMP n … n Papers, videos and demos: http://www.intuilab.com/~dragice 99