Post-Wimp Interaction With Desktop

New Tools and Techniques

Pierre Dragicevic

1 WIMP Interaction

n WIMP = Windows, Icons, Menus, Pointer

+

n Standard Devices: n Standard Interaction Techniques:

n Mouse n Windows, widgets, n Keyboard n Text input, click, double-click, d&d 2 What Next and Why?

n WIMP system ( Star, 1979)

3 Post-WIMP Interaction Paradigms

Better exploit our motor abilities:

n Bimanual Interaction

n « Multifinger » Interaction

Map Zooming [Hinckley et al 98]

4 SmartSkin [Rekimoto 2002] Post-WIMP Interaction Paradigms

Better exploit our perceptual abilities:

n Information visualization

n Enhanced displays

TreeMaps [Schneiderman et al]

Multiple screens & large displays 5 Post-WIMP Interaction Paradigms

Make interaction more « direct »:

n Gestural interaction

n See-through tools

Toolglasses [Bier et al 93] 6 Two Research Directions n Explore Post-WIMP interaction further:

n Propose innovative solutions to known problems n Investigate alternative interaction paradigms

n Disseminate existing research:

n Integrate Post-WIMP paradigms into GUI toolkits n Provide new design guidelines

7 My Research n Explore Post-WIMP interaction further:

n Propose innovative solutions to known problems n Investigate new interaction paradigms

n Disseminate existing research:

n Integrate Post-WIMP paradigms into GUI toolkits n Provide new design guidelines n Support the exploration of new approaches:

n Study more usable and more flexible tools n Advocate a vision of Post-WIMP with sound examples 8 My Research

New tools

New techniques

9 Introduction

I. Input Configurator

New tools II. Artistic Resizing

III. SpiraClock

New techniques IV. Fold n’ Drop

Conclusion 10 Introduction

I. Input Configurator I. Input Configurator

II. Artistic Resizing

An Input Interaction Model for III. SpiraClock Highly Configurable Multi-Device Interactive Systems

IV. Fold n’ Drop

Pierre Dragicevic Jean-Daniel Fekete Conclusion 11 Goal: Support Alternative Input

n WIMP system n Input-Open System (Xerox Star, 1979)

12 Why Using Alternative Input? n Augmenting the bandwith n Optimizing interaction according to:

the task the user the environnement 13 Current State of the Art n Very poor support for alternative input devices in:

n Current operating systems n Current applications n Current toolkits n Little research on input management n Post-WIMP tools are specialized towards:

n A fixed set of input devices n A fixed set of interaction techniques n No approach deals with:

n Multiple and heterogeneous input (enriched / impoverished)

n Flexibility and configurability 14 The Input Configuration Model n Approach:

n Fully separate physical devices from applications

n Provide a flexible way to map devices to applications

15 The Input Configuration Model

n Device:

n Parametrizable black box n Processes input into output

n I/O Slots: Input Output n Simple types slots slots n + compound types

n Implicit I/O:

n User input Implicit Implicit input output n Feedback

16 The Input Configuration Model n Device types:

n System (hardware ressources) Input Configuration n Adapter (software processes) n Application (specific to each app) n Input Configuration:

n A set of system and application devices connected through adapter devices

System Adapter Application 17 devices devices devices The ICON Toolkit n System Devices:

18 The ICON Toolkit

n Adapter devices:

n Data processing

19 The ICON Toolkit

n Adapter devices:

n Data processing n Graphical feedback

20 The ICON Toolkit

n Adapter devices:

n Data processing n Graphical feedback n Interaction techniques

21 The ICON Toolkit

n Application Devices:

n Example: IConDraw tools

22 The ICON Toolkit

n Graphical Toolkit Devices:

n Example: Swing devices

23 The ICON Toolkit

n The input configuration editor:

EditingEditing panepane

AvailableAvailable devicesdevices

24 The ICON Toolkit

n The input configuration editor

25 The ICON Toolkit n The input configuration editor

n Dynamic wire drawing algorithm

26 The ICON Toolkit n The input configuration editor

n Fluid expand/collapse

27 The ICON Toolkit n The input configuration editor

n Fisheye-drag

28 Examples

1. Exploiting rich input

n Pressure-sensitive drawing inside IConDraw

29 Examples

1. Exploiting rich input

30 Examples

1. Exploiting rich input

n Symmetric bimanual interaction in IConDraw

31 Examples

1. Exploiting rich input

32 Examples

1. Exploiting rich input

n Integral control of the Responsive Face [Perlin 97]

33 Examples

1. Exploiting rich input

n Integral control of the Responsive Face [Perlin 97]

34 Examples

1. Exploiting rich input

35 Examples

2. Improving Accessibility

n Keyboard ? Pointing

2nd order control 36 Examples

2. Improving Accessibility

n Pointer ? Text editing

« Floating QuikWriting » [Perlin] 37 Examples

2. Improving Accessibility

n Pointer ? Text editing

« Floating QuikWriting » [Perlin] 38 Examples

3. Augmenting WIMP applications

n « Augmented pointing »

Filter Events

39 Examples

3. Augmenting WIMP applications

n « Augmented pointing » : smoothed

40 Examples

3. Augmenting WIMP applications

n « Augmented pointing » : throwable cursor

41 Examples

3. Augmenting WIMP applications

n « Augmented pointing » : speech cursor

rightNoiseNoise…… right releasereleasedowndown slowerslower presspressstopstopupup

42 Examples

3. Augmenting WIMP applications

n « Generalized Drag and Drop »

43 Projects using ICON

n Marina II (2001) Photograph-based architectural modelling tool

n Svalabard [Huot et al 2003] Post-WIMP sketching-based 3D modelling interface

n MaggLite [Huot et al 2004] Post-WIMP UIMS based on ICON

n LRI, University of Orsay [Appert et al 2003] Experimental comparison of WIMP / Post-WIMP techniques

n Smart Home and Impairment (INT/ENST/EDF 2005) Configurable software architecture for accessibility

n Student projects 44 Introduction

I. Input Configurator II. Artistic Resizing

II. Artistic Resizing

A Technique for Rich

III. SpiraClock Scale-Sensitive Vector Graphics

Pierre Dragicevic IV. Fold n’ Drop Stéphane Chatty David Thevenin Jean-Luc Vinot Conclusion 45 Graphically Rich User Interfacs

n Graphic designers are given more and more importance in GUIs

n Exploitation of vector graphics (e.g, Scalable Vector Graphics)

n IntuiKit [Chatty et al 2004]

46 IntuiKit: Example of Team Work

Programming

Low-fi prototyping Integration

Graphic design

47 The Resizing Problem

n Fixed size

n Naive scaling

n Common resizing

48 Expressing Resizing n Resizing is commonly described using formulae

w

xL · x = (w-w ) / 2 · w = 5 r L L B yL · yL = (h-hL) / 2 · hB = 5 h hL · wL = 20 · hL = 10 · r = 20 wL hB

wB n These formulae are:

n Translated into code by the programmer n Or used as an input to constraint-solving systems

49 Expressing Resizing n But designers think visually n They are used to produce visual variants

On white On black On red Medium Small paper paper paper size size

n Variants are also useful to convey general laws

50 Artistic Resizing

1. Designers produce variants 2. IntuiKit interprets using their authoring tool the example set

51 Artistic Resizing A scenario

52 Artistic Resizing A scenario

53 Artistic Resizing A scenario

54 Artistic Resizing A scenario

55 Artistic Resizing A scenario

56 Artistic Resizing How does it work? n Based on local interpolation of transformations

T1 T1’

57 Artistic Resizing How does it work? n Each variant of T1 is associated with the example’s bounding box

T1 T1’ ?

T1’’ T1’’’

58 Artistic Resizing Orthogonal Interpolation n Orthogonal Interpolation hypothesis:

n Width only impacts the X contribution n Height only impacts the Y contribution

11 a12 a13 width X a X0

21 a22 a23 height Y = a Y0

1 0 0 1 1

59 Artistic Resizing Orthogonal Interpolation n Piecewise linear interpolations are performed along width and height separately:

height a23 a22 a21

a11a12 a13

width

60 Artistic Resizing Properties of Orthogonal Interpolation

n Preserves algebraic measures:

n If horizontal or vertical distances are the same on the examples they will remain the same

61 Artistic Resizing Properties of Orthogonal Interpolation

n Preserves relative ratios:

n Ratios that are invariant on the examples will remain invariant

62 Artistic Resizing Properties of Orthogonal Interpolation

n Preserves contacts:

n Two points that coincide on the examples will always coincide

63 Artistic Resizing Properties of Orthogonal Interpolation

n Preserves parallelism:

n Two lines that are parallel on the examples will remain parallel

64 Artistic Resizing

n Supports invariants but also finely tuned behaviors

w

xL r yL h hL

wL hB

wB · xL = (w-wL) / 2 · wB = 5 · yL = (h-hL) / 2 · hB = 5 · wL = 20 · hL = 10 · r = 20 65 Artistic Resizing Blending with traditional layout managers

n Containment hierarchies:

66 Artistic Resizing Blending with traditional layout managers

n Inside an « aqua-dock » :

67 Artistic Resizing vs. Constraint Inference n Demonstrational layout specification systems:

n Peridot [Myers 90] n Chimera [Kurlander & Feiner 93]

n Constraint inference: n Orthogonal interpolation:

n Extensive search n Ensures the preservation of for invariants « interesting » invariants n Sensitive to combi- n Fast computation, even with natorial explosion complex vector graphics n Over-constrained or n Limitations are known, unintended rules results are easy to predict 68 Introduction III. SpiraClock I. Input Configurator

II. Artistic Resizing

A Continuous and Non-Intrusive III. SpiraClock Technique for Upcoming Events

IV. Fold n’ Drop Pierre Dragicevic Stéphane Huot

Conclusion 69 Time Management Tools

n Calendars displays:

n Good for long-term time n Reminders and management Alarms:

n But useless for n Prevent forgetting upcoming events n But too intrusive n Don’t allow continuous anticipation

70 SpiraClock

n Analog desktop clock

n Events are displayed as sectors

71 SpiraClock

n Event are aligned with minute marks

n Relative times can be read directly

72 SpiraClock

n One example: viewing bus departures

It’s 18h50. What buses can I catch ?

73 SpiraClock

n One example: viewing bus departures

74 SpiraClock

n One example: viewing bus departures

75 Uses of SpiraClock n BusClock used in Ecole des Mines de Nantes Since 2002

n Other applications:

n ShuttleClock - shuttles of University of Maryland n OutClock - synchronized with MS Outlook n CalendarClock - uses VCal exchange format 76 Introduction IV. Fold n’ Drop I. Input Configurator

II. Artistic Resizing

Combining Paper-Based and III. SpiraClock Crossing-Based Interaction Paradigms for Dragging and Dropping between Overlapping Windows IV. Fold n’ Drop

Pierre Dragicevic

Conclusion 77 Drag & Drop Scenario n Moving a file into a folder

78 Drag & Drop Scenario n Problem: the target is hidden

79 Drag & Drop Scenario n Solution 1: adjust windows

adjust

drag & drop

80 Drag & Drop Scenario n Solution 2: cut & paste

cut

bring to the front

81 Drag & Drop Scenario n Solution 2: cut & paste

paste

82 Using Navigation n Dual representation ® Mapping problems (XP taskbar)

n Activation costs (mouse-over timeout on Exposé)

83 Using Window Navigation n Activation is a superfluous concept for D&D tasks

84 The Fold n’ Drop Technique n Folding a window

n The mouse leaves the window

n The mouse leaves then returns back

85 The Fold n’ Drop Technique n Other gestures

n Leafing through

n Discarding a window

n Unfolding

86 Video

87 Fold n’ Drop builds upon…

n Paper-based window metaphors

n The « Folded » effect [Beaudouin-Lafon 01]

Ametista Toolkit [Roussel 03] AttrActive Windows [Denoue 03] 88 Fold n’ Drop builds upon… n Gestural interfaces

n Natural, « physical » gestures

n Do not need a recognition engine

n Naturally blend with direct manipulation [see e.g., Pook et al 00]

n Little interference with other interactions

89 Fold n’ Drop builds upon… n The crossing-based interaction paradigm [Accot & Zhai 02]

n Fn’D uses double-crossing and n-crossing

n Extends the paradigm:

• Point & Click ® Cross boundaries • Drag ® Push boundaries • Double-Click ® Timed double-crossing

n Timing is made visible:

90 Introduction

I. Input Configurator Conclusion II. Artistic Resizing

III. SpiraClock

IV. Fold n’ Drop

Conclusion 91 Conclusion n Common desktop computing activities can still be improved:

n Management of multiple tasks • Task switching • Data transfer between applications n Management of personal data • Viewing, searching • Organizing n Time management • Long-term • Short-term n But how to design Post-WIMP techniques ? 92 Conclusion Post-WIMP GUI Design as a Creative Activity n The success of a technique is difficult to predict:

n Efficiency + pleasingness ? Importance of « trial and error » (iterative design) n But techniques also require fine tuning:

n Timings, input filtering, graphical effects, color design,… ? Need for closer experimentation-evaluation loops n « Creative tools » allow to explore and test by ourselves:

n Flexible, open n High degree of control, finely tunable n Fast experimentation-evaluation loop n Simple and predictable

93 Conclusion 5 characteristics of Post-WIMP Techniques

1. Specificity 2. High I/O bandwidth

n Multiple input channels n Advanced output devices or rich graphics 3. Directness

n Stimulus/Response compatibility 4. Reactivity and Fluidity

n Responsive and continuous feedback 5. Straightforwardness

n Familiar metaphors n Simple, predictable system behavior 94 Conclusion About Predictability n Example: gestures in Fold n’ Drop

« I missed the fold »

« I returned back too late » 95 Conclusion About Predictability n Another example: freehand polyline drawing

n Using a smart tool (stroke beautifier):

96 Conclusion About Predictability n Another example: freehand polyline drawing

n Using a smart tool (stroke beautifier):

97 Conclusion About Predictability n Another example: freehand polyline drawing

n Using a dumb tool (spring-loaded cursor):

98 Conclusion Future Work n Continue to explore new interaction paradigms

n Desktop n Non-desktop n Generalize and theorize:

n Intelligence vs predictability n Explicit gestures n Augmented WIMP n … n Papers, videos and demos: http://www.intuilab.com/~dragice 99