User Interface Toolkits Graphical Libraries

Master Informatique - Université Paris-Sud

Outline

Software layers

User interface toolkits Graphical libraries

Window systems Michel Beaudouin-Lafon Laboratoire de Recherche en Informatique User interface toolkits Université Paris-Sud / CNRS [email protected] http://insitu.lri.fr Applications frameworks

Interface builders

Software layers Output devices

Bitmap screens Application CRT, LCD, Plasma, … Framework Spatial resolution: about 100dpi Color resolution (« color depth ») : User interface toolkit B&W, grey levels, color table, direct color

Window system

Graphical library R G B Operating system Device drivers Temporal resolution: 10 to 100 frames per second Bandwidth: 25 img/s * 1000x1000 pixels * 3 bytes/pixel = 75 Mb/s GPU : Graphics Processing Unit

Input devices Graphical libraries

2D input devices Drawing model Mouse, Tablet, Joystick, Trackball, Touch screen Direct drawing (painter’s algorithm) Type of user control Structured drawing: scene graph position, motion, force, … ; linear, circular, … Edit the data structure Mapping of input dimensions position, speed, acceleration Graphical objects are defined by: transfer function (gain) Their geometry Motor space vs. visual space Their graphical attributes separate or identical color, texture, gradient, transparency, lighting

Other input devices Graphical libraries Keyboards, Button boxes, Sliders Direct drawing: Xlib, Java2D, OpenGL 3D position and orientation sensors Structured drawing: Inventor (3D), SVG Simulated devices

Managing input in an interactive system Event-driven programming

Query Sampling Events while running do wait until event queue not empty // blocking Blocking Busy waiting Event queue ev := first event from queue // extract event target := findTarget(ev) if target ≠ NIL then target.handleEvent(ev) end while

Very different from traditional algorithmic approach

Window systems Windowing models

Organize display space in independent areas Tiling Hierarchical Resource sharing

Window = autonomous area on the screen - for display - for input (event dispatching) Overlapping Virtual screens

Window management User interface: « window manager » Application programming interface

Window systems Client-server architecture

Drawing model Virtual terminals Redraw hidden parts Clients are independent of each other Example : X Window System Input management Demultiplex event across applications Client Concept of « focus » New events Window system: Client request redraw, create/delete window Input devices: focus changes, cursor enters/leaves window Client Server (network)

Client-server architecture Architecture client-serveur

requests multiplexing display requests multiplexing display

events demultiplexing input

Client Client

Client Client Server Server (network) (network)

User interface toolkits Widget layout

Abstraction : the widget General rules Interactive object, component A widget is geometrically enclosed in its parent Button, menu, scrollbar, dialog box, … The parent controls the layouts of its children

Layout algorithm file Natural size of each child A widget = three facets Final size and positions Cancel OK imposed by the parent Presentation – Behavior – Application interface Constraints : Grid, form, etc. file Interface = widget tree Nodes: containers (windows, menu bar, dialog box, …) Cancel OK Leaves: simple widgets (buttons, scrollbars, …) Dynamic layout

Facets of a widget Application interface: callback functions

Presentation 1. Registration of callback when widget is created Visual appearance Configurable (« resources ») DoSave (…) { … } 2. Callback function is called when widget is activated Behavior Reaction to user actions DoSave (…) { … } Non configurable (or very limited) Problem: « spaghetti » of callbacks Sharing state among widgets and callbacks using global variables Application interface Notification of state changes global string filename; DoSetFile () {filename = …}

DoSave () { SaveTo(filename) }

Application interface: active values Application interface: message passing

Bi-directional link between a state variable of the widget and An object is associated to a widget, a variable of the application its methods are called when a state of the widget changes 0 i saveDialog

26 i saveDialog.Clicked(event) i := 12 12 i Better encapsulation saveDialog { string filename }

Problems Advantages saveDialog.EditField(event) Limited to simple data types Multiples views { this.filename := … } Back link (widget to app) can be costly saveDialog.OK(event) { DoSave (this.filename) }

User interface toolkits Application frameworks

Many available toolkits Application skeleton Xt, Motif – historical (X Windows) Incomplete code: general structure of theapplication Qt, GTK – Linux Includes what is not supported by the toolkit AWT, Swing – Java Global structure of the application Tck/Tk – multi-plateformes [active values] Global functions (history, copy-paste, …) Non-widget interaction (e.g., drag-and-drop) Many limitations Shows the limitations of the programming language Programming is cumbersome Interaction limited to the interior of the widget Example : MacApp (Apple, 1986) example : no drag-and-drop Concept of a document (content of a window) Limited extensibility : adding new widgets types is difficult Concept of action (that can be done and undone)

Interface builders Interface builders

Description (text or graphics) of part of the interface Generating the final application Generation of a runnable version

Run-time module Run-time module Editor Functional core Functional core of interface builder compile of interface builder

linking Compiled interface linking

load Interactive Interactive at run-time application application interface description

Interface Builder Conclusion

NeXT, then Apple Advantages of these tools Reduce development and maintenance costs Facilitate compliance with style guides

Limitations of these tools Interaction style based on widgets Limited extensibility Difficult to program non-standard interactions

Research issues Beyond the widget model Define better languages and environments