Lecture 1 A Programming Approach to HCI Tue, Apr 2, 2002 CS377A • Spring Quarter 2002 Jan Borchers, Stanford University http://cs377a.stanford.edu/

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 1 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 2

Questionnaire Welcome to CS377A!

We will start with the course in a few minutes.
Instructor: Jan Borchers
Meanwhile, please complete the questionnaire; it
Acting Assistant Professor, Interactivity Lab helps us with admission if necessary, and lets us tune
Research: Post-desktop and multimedia UIs (Personal the course to your background. Orchestra,...)
Teaching: HCI since '96, CS247A (3x), CS377C
PhD CS Darmstadt, MS,BS CS Karlsruhe&London
TA: David Merrill
MSCS student, BS SymSys
Research: music & speech interfaces, facial animation
Teaching: TA since '99 (CS147, CS108, teaching CS193J)

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 3 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 4 Administrative Information Course organization

Course times: Tue+Thu 9:15-10:45
Requirements: CS147, Java (CS193J or equivalent)
Course location: Meyer 143
Enrollment: limited to 20 students due to project-
Others as announced via mailing list oriented course; we will email you within 24hrs
Course home page: http://cs377a.stanford.edu/
Credits: 3 (Letter or CR/NC)
Mailing list: Automatic upon registering for the course
Substitutes CS247A requirement this academic year
Email: Please use [email protected]
Grading: which reaches both David and Jan
Lab project assignments throughout Quarter (80%)
Jan's Open Office: Tue 3:30-4:30, Gates 201
Final exam (20%)

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 5 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 6

Course Topic Course Topic

What makes a UI tick?
Part III: UIs Beyond The Desktop
Technical concepts, software paradigms and
Think beyond today's GUI desktop metaphor technologies behind HCI and user interface
E.g.: UIs for CSCW, Ubicomp development
The Lab
Part I: Key concepts of UI systems
Part I: Implementing Simple Reference Window System
Window System Architecture Model
Part II: Development using several existing GUI toolkits
Part II: Review and comparison of seminal systems (such as Java/Swing, InterfaceBuilder)
Smalltalk, Mac, X/Motif, AWT/Swing, NeXT/OS X,…
Part III: Working with Stanford's Interactive Room OS
Paradigms & problems, design future UI systems
Overview of UI prototyping tools, with focus on Tcl/Tk

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 7 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 8 Assignment 0: Reading Assignment "Hello Window System"
Use the GraphicsEventSystem library to implement a
For Thursday, please read the following article minimalistic window system
Stuart K. Card, Jock D. Mackinlay and George G.
Creates and displays empty background (desktop) on Robertson: "A morphological analysis of the design space of input devices", ACM Transactions on Information Systems, the screen 9(2), 99-122, 1991
In-class exercise
Available from the ACM Digital Library
Work in groups as needed (http://www.acm.org/dl/ - Stanford has a site license) or the course home page
Instructions: see assignment
Submit via upload by end of class, or by midnight if you cannot finish it in class

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 9 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 10

Review

3-Part course structure
I: Theory, II: Systems, III: Future Lecture 2
Accompanying lab assignments
Started Simple Reference Window System
Thu, Apr 4, 2002 Register in Axess for mailing list
http://cs377a.stanford.edu/

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 11 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 12 CS377A & HCI A Brief History of User Interfaces

247B
Batch-processing
No interactive capabilities 448
All user input specified in advance (punch cards, ...) 378 Cog M250
All system output collected at end of program run 277 Hap (printouts,...)
-> Applications have no user interface component 377B Cog distinguishable from File I/O 147 547 377A
Job Control Languages (example: IBM3090–JCL, anyone?): specify job and parameters 377D Eth 247A

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 13 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 14

A Brief History of User Interfaces A Brief History of User Interfaces
Time-sharing Systems

Command-line based interaction with simple terminal Menu-based systems

Shorter turnaround (per-line), but similar program structure Discover "Read & Select" over "Memorize & Type" advantage
-> Applications read arguments from the command line,
return results Still text-based!

Example: still visible in Unix commands Example: UCSD Pascal Development Environment

Full-screen textual interfaces -> Applications have explicit UI component
But: choices are limited to a particular menu item at a time
Shorter turnaround (per-character) (hierarchical selection)
Interaction starts to feel "real-time" (example: vi)
-> Application still "in control"
-> Applications receive UI input and react immediately in main "loop" (threading becomes important)

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 15 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 16 A Brief History of User Interfaces Design Space of Input Devices
Graphical User Interface Systems

From character generator to bitmap display (Alto/Star/Lisa..) Card, Mackinlay, Robertson 1991

Pointing devices in addition to keyboard Goal: Understand input device design space -> Event-based program structure
Insight in space, grouping, performance reasoning, new
Most dramatic paradigm shift for application development design ideas
User is "in control"
Idea: Characterize input devices according to
Application only reacts to user (or system) events physical/mechanical/spatial properties
Callback paradigm
Morphological approach
Event handling
device designs=points in parameterized design space
Initially application-explicit
combine primitive moves and composition operators
Later system-implicit

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 17 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 18

Primitive Movements Radio Example
Input device maps physical world to application logic
Input device :=
Manipulation operator P, dP R, dR
Input domain F, dF T, dT
Device State
Resolution function In->Out
Output domain
Additional work properties

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 19 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 20 Composition Design

Merge Space
Result=cross product (partial
E.g., mouse x & y
Layout viz.!)
Spatial collocation
E.g., mouse xy & buttons Complete space
How different from merge? := {all possible
Connect
combinations of Chaining primitives and
E.g., mouse output & cursor
composition Virtual devices operators}. Mouse=1 point! CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 21

In-Class Group Exercise: Is This Space Complete? Lightning II
Place the Lightning II
No – it focuses on mechanical movement infrared baton system into
the design space Voice

Two batons in user's hands, Other senses (touch, smell, ...) 1 tracker in front of user
But: Already proposes new devices
Batons can be moved in
Put circles into the diagram and connect them space freely, but only horizontal and vertical position are detected with 7 bit accuracy (not distance from tracker)
Each baton has an on/action button and an off button

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 23 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 24 Testing Points Effectiveness
How well can the intention be communicated?
Evaluate mappings according to
Various figures of merit possible
Expressiveness (conveys meaning exactly)
Performance-related
Effectiveness (felicity)
Device bandwidth (influences time to select target,
Visual displays easily express unintended meanings ergonomics and cognitive load)

For input devices, expressiveness suffers if |In|≠|Out| Precision
Error (% missed, final distance, statistical derivatives)
|In|<|Out|: Cannot specify all legal values
Learning time
|In|>|Out|: Can specify illegal values
Mounting / grasping time
Pragmatic
Device footprint, subjective preferences, cost,...

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 25 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 26

Example: Device Footprint Assignments

Circle size:=device footprint
For Tuesday:
Black: with 12" monitor
Read Window System Architecture chapter from Gosling's
White: with 19" monitor NeWS book (James Gosling, David S. H. Rosenthal, and
What do we see? Michelle J. Arden, "The NeWS Book", Springer-Verlag,
Tablet, mouse expensive 1989, Chapter 3; see paper handout)
Worse with larger displays
For Thursday:
But:
Implement basic Window class (see assignment handout)
Mouse Acceleration alleviates this (model of C:D ratio?)
Higher resolution mice

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 27 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 28 Window Systems: Basic Tasks

Basic window system tasks:
Input handling: Pass user input to appropriate application Lecture 3
Output handling: Visualize application output in windows
Window management: Manage and provide user controls for windows Tue, Apr 9, 2002
This is roughly what our Simple Reference Window System will be implementing

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 29 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 30

In-Class Exercise: Window Systems: Requirements Window Systems Criteria
Independent of hardware and operating system
In groups of 2, brainstorm criteria that you would look
Legacy (text-based) software support (virt. terminals) at when judging a new window system

No noticeable delays (few ms) for basic operations We will compile the answers in class afterwards (edit text, move window); 5+ redraws/s for cursor
Customizable look&feel for user preferences
Applications doing input/output in parallel
Small resource overhead per window, fast graphics
Support for keyboard and graphical input device
Optional: Distribution, 3-D graphics, gesture, audio,...

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 31 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 32 Window Systems: Criteria Window Systems: Criteria

Availability (platforms supported)
Appearance (Look & Feel, exchangeable?)
Productivity (for application development)
Extensibility of WS (in source code or at runtime)
Parallelism
Adaptability (localization, customization)
external: parallel user input for several applications possible
At runtime; e.g., via User Interface Languages (UILs)
internal: applications as actual parallel processes
Resource sharing (e.g., fonts)
Performance
Distribution (of window system layers over network)
Basic operations on main resources (window, screen, net),
API structure (procedural vs. OO) user input latency – up to 90% of processing power for UI
API comfort (number and complexity of supplied
Graphics model (RasterOp vs. vector) toolkit, support for new components)

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 33 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 34

Window Systems: Criteria Window Systems: Conflict

Independence (of application and interaction logic
WS developer wants: elegant design, portability inside programs written for the WS)
App developer wants: Simple but powerful API
IAC (inter-application communication support)
User wants: immediate usability+malleability for
User-initiated, e.g., Cut&Paste experts Technique Selection Clipboard DDE OLE  Partially conflicting goals Duration short short medium long  Architecture model shows if/how and where to solve Data types special special special any  Real systems show sample points in tradeoff space Directed yes no yes no Relation 1:1 m:1:n 1:1 m:n Abstraction low low medium high

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 35 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 36 The 4-Layer Model The 4-Layer Model of Window System Architectures of Window System Architectures
Layering of virtual Applications
UI Toolkit (a.k.a. Construction Set) machines
Offers standard user interface objects (widgets)
Good reference model User Interface Toolkit
Window Manager

Existing (esp. older) Window Manager Implements user interface to window functions systems often fuzzier
Base Window System
Where is the OS? Base Window System
Provide logical abstractions from physical resources (e.g.,
Where is the user? windows, mouse actions) Graphics & Event Library

physical vs. abstract Graphics & Event Library (implements graphics model)
communication Hardware high-performance graphics output functions for apps,
cf. ISO/OSI model more abstract, application-/user-oriented register user input actions, draw cursor

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 37 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 38

In-Class Exercise: Map our A Note On Gosling's Model Window System into model
Same overall structure
Which layers are supplied by the toolkit?
But certain smaller differences
Which layers are you implementing?
E.g., defines certain parts of the GEL to be part of the BWS
What is missing so far?
Written with NeWS in mind
We will follow the model presented here
More general
5 years newer
Includes Gosling's and other models

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 39 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 40 Graphics & Event Library The RasterOp Model Apps Graphics objects UITK Event queues
Original graphics model WM & actions
BWS Logical coordinates Canonical events Device-independent Suited to bitmap displays with linear video memory GEL
Adresses individual pixels directly HW Memory addresses Driver-specific data Device-dependent
Fast transfer of memory blocks (a.k.a. bitblt: bit block transfer) Graphics hardware Device drivers
Absolute integer screen coordinate system

Device-dependent sublayer to optimize for hardware Resolution problem

Device-independent sublayer hides HW vs. SW Simple screen operations (the XOR trick,...)
implementation (virtual machine) But break down with color screens

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 41 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 42

The Vector Model Graphics Library Objects: Canvas

API uses normalized coordinate system
Memory areas with coordinate system and memory-
Device-dependent transformation inside layer to-pixel mapping
Advantage: units are not pixels of specific device anymore
Defined by: Start address, size, bit depth, logical
Applications can output same image data to various screens arrangement in memory (only relevant for pixmaps) and printer, always get best possible resolution (no
Z format (consecutive bytes per pixel, easy pixel access) "jaggies")
XY format (consecutive bytes per plane, easy color access)
Originally implemented using Display PostScript
Included arbitrary clipping regions  a.k.a. "Stencil/Paint Model" Z format XY format

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 43 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 44 Graphics Library Objects: Graphics Library Objects: Output Objects Graphics Contexts
Elementary
Status of the (virtual) graphics processor
Directly rendered by graphics hardware

E.g., Circle, line, raster image Bundle of graphical attributes to output objects

Complex E.g., line thickness, font, color table

Broken down by software into elementary objects to render Goal: reduce parameters to pass when calling graphics
Example: Fonts operations
Broken down into raster images (bitmap/raster/image font,
Not always provided on this level quick but jagged when scaled)
Or broken down outline curves (scalable/outline/vector fonts, scalable but slower)
Real fonts do not scale arithmetically!

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 45 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 46

Graphics Library: Actions Graphics Library: Actions
Output (Render) actions for objects described above
Who has to do redraw?
Three "memory modes"
Buffered modes: GEL can redraw, needs trigger
Direct/Immediate Drawing
Immediate mode: application needs to redraw (may
Render into display memory and forget implement buffer or display list technique itself)
Command-Buffered/Structured Drawing, Display List Mode
Mouse cursor is always redrawn by GEL (performance)
Create list of objects to draw
Unless own display layer for cursor (alpha channel)
May be hierarchically organized and/or prioritized
Triggered by event part of GEL
Complex but very efficient for sparse objects
Clipping is usually done by GEL (performance)
Data-Buffered Drawing
Draw into window and in parallel into "backup" in memory
Memory-intensive but simple, efficient for dense objects

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 47 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 48 Event Library: Objects Event Library: Actions

Events
Drivers deliver device-specific events interrupt-driven
Driver-specific: physical coordinates, timestamp, device- into buffers with timestamps specific event code, in device-specific format
EL cycles driver buffers, reads events, puts unified
Canonical: logical screen coordinates, timestamp, global events into 1 queue per device (all queues equal event code, in window system wide unified format format)
Event Library mediates between mouse/kbd/tablet/... drivers
and window-based event handling system by doing this Update mouse cursor without referring to higher unification layers
Queue
EL offers one event queue per device

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 49 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 50

GEL: Extensions GEL: Summary

GL: Offer new graphics objects/actions (performance)
4-layer model
EL: Support new devices
Graphics & Event Library
Availability
Hides hardware and OS aspects

Most systems: Not accessible to application developer Offers virtual graphics/event machine

GEL as library: Only with access to source code (X11) Often in same address space as Base Window System

GEL access via interpreted language: at runtime (NeWS) Many GEL objects have peer objects on higher levels

Example: Download PostScript code to draw E.g., windows have canvas triangles, gridlines, patterns,... into GEL

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 51 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 52 In-Class Design Exercise

Work in groups of 2
Define XWindow class (basic components+methods) Lecture 4
Integrate with WindowSystem class
Windows as components of WindowSystem?
Windows as first-class objects? Thu, Apr 11, 2002
Think about future needs (defining repaint() methods,...)
No user interface to windows yet; will be WM
Time to finish until Thursday's class

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 53 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 54

Review Base Window System: Tasks

Apps
Criteria for judging window systems UITK
Provide mechanisms for operations on WS-wide data
4-layer architecture WM structures BWS

Graphics & Event Library GEL Ensure consistency
Device-dependent & independent sublayer HW
Core of the WS
Objects: Canvas, output objects, graphics contexts
Most fundamental differences in structure between
Graphics models, drawing modes different systems
Canonical events
user process with GEL, part of OS, privileged process
Extensibility
In general, 1 WS with k terminals, n applications, m objects (windows, fonts) per app (l WS if distributed)

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 55 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 56 Base Window System: Structure Base Window System: Objects Requests, Output, Dialog input, Changes State messaging for apps 1..n
Windows, canvas, graphics contexts, events Apps Access Control Addressing Connection Mgmt.
UITK Requested explicitly from applications (except WM Request Demultiplex Resource Operations events), but managed by BWS—why? BWS Mutual Exclusion Multiplex Synchronization
Manage scarce resources for performance & efficiency GEL Memory Allocation Queue/Dequeue
HW Elementary op's. Applications share resources Canvas Events Objects
Consistency and synchronization Graphics Library Event Library
Real vs. virtual resources
(Video) memory, mouse, keyboard, usually also network
Applications only see "their" virtual resources

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 57 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 58

Windows & Canvas Events
Components:
Components:
Owner (application originally requesting the window)
Event type App 1...App m
Users (reference list of IDs of all applications temporary
Time stamp aiming to work with the window)
Type-specific data Order
Size, depth, border, origin
Location
State variables (visible, active,...)
Window
Device 1...Device n
Canvas Application

=Window without state; not visible Event Processing:

Collect (multiplex) from device queues Operations:
Order by time stamp, determine application & window
Drawing in application coordinate system
Distribute (demultiplex) to application event queues
State changes (make (in)visible, make (in)valid,...)

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 59 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 60 Events Fonts

BWS can generate events itself based on window
Increasingly offered by GEL (performance), but states (e.g., "needs restoring") or certain incoming managed here event patterns (replace two clicks by double-click),
Load completely into virtual memory, or and insert them into queue
Load each component into real memory, or
Load completely into real memory
Components
Application owner, other apps using it (as with windows)
Typically shared as read-only -> owner "just another user"
Name, measurements (font size, kerning, ligatures,...)
Data field per character containing its graphical shape

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 61 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 62

Graphics Context Color Tables

Graphics Context Components
Components
Owner app, user apps
Owner app, user apps
Graphics attributes (line thickness, color index, copy
Data fields for each color entry function,...)
RGB, HSV, YIQ,...
Text attributes (color, skew, direction, copy function,...)
Color table reference
Fault tolerance
GEL: 1 Graphics context at any time, BWS: many
BWS should hold defaults for all its object type parameters
Only one of them active (loaded into GEL) at any time to allow underspecified requests
BWS should map illegal object requests (missing fonts,...) to legal ones (close replacement font,...)

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 63 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 64 Communication Bandwidth BWS: Actions

WS needs to talk to other apps across network
Typically on top of ISO/OSI layer 4 connection (TCP/IP,...)
Basic set of operations for all object types
But requires some layer 5 services (priority, bandwidth,...)
Allocate, deallocate
Usually full-duplex, custom protocol with efficient coding
Other elementary operations for certain types
Exchange of character and image data, often in bursts
Read and write events to and from event queues
Each application expects own virtual connection
Filtering events for applications  Bandwidth is scarce resource

Components of a Connection object: How to manage window collection in BWS?

Partner (IP+process,...), ID, parameters, encoding, message class Tree (all child windows are inside their parent window) (priority,...)
Why? ->Visibility, Event routing
Elementary operations: decode, (de)compress, checksum,...
Remember: on the BWS level, all UI objects are windows
Optional operations: manage connection, address service —not just document windows of applications!

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 65 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 66

In-Class Exercise Shared Resources

Determine a valid tree structure for the window
Reasons for sharing resources: Scarcity, collaboration arrangement shown below
Problems: Competition, consistency
1 Solution: Use "users" list of objects
Add operations to check list, add/remove users to object 5 2
Deallocate if list empty or owner asks for it 6 8 3
7 How does BWS handle application requests?
4 Avoid overlapping requests through internal synchronization
Use semaphores, monitors, message queues

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 67 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 68 Synchronization Options OS Integration
Single address space
Synchronize at BWS entrance
No process concept, collaborative control (stability?)
One app request entering the BWS is carried out in full
"Window multitasking" through procedure calls before next request is processed (simple but potential delays) (cooperation on common stack)
Synchronize on individual objects
Xerox Star, Apple Mac OS, MS Windows 3.x
Apps can run in parallel using (preemptive) multitasking
BWS in kernel
Operations on BWS objects are protected with monitors
Apps are individual processes in user address space
Each object is monitor, verify if available before entering
BWS & GEL are parts of kernel in system address space
high internal parallelism but complex, introduces overhead
Each BWS (runtime library) call is kernel entry (expensive but handled with kernel priority)
Communication via shared memory, sync via kernel

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 69 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 70

OS Integration Summary

BWS as user process
BWS works with device- and OS-independent
BWS loses privileges, is user-level server for client apps, abstractions (only very general assumptions about OS) Communication via Inter-Process Communication (IPC)
Supports system security and consistency through
Single-thread server ("secretary"): no internal parallelism, encapsulation and synchronization sync by entry
map n apps with virtual resource requirements to 1 hardware
Server with specialized threads ("team"): each thread handles specific server subtask, shared BWS objects are
Offers basic API for higher levels (comparable to our protected using monitors Simple Reference Window System)
Multi-server architecture: Several separate servers for
Where are window controls, menus, icons, masks, ...? different tasks (font server, speech recognition and synthesizing server,... — see distributed window systems)

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 71 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 72 Assignment #2

Extend the Simple Window System to actually create visible windows and close them again
Include sample application that creates three Lecture 5 overlapping windows, drawing different geometrical shapes inside each after creating it, and then closes Tue, Apr 16, 2002 them again one by one. Make the app pause between each creation and closing so it becomes clear that the redrawing of uncovered windows happens correctly.
See assignment handout for more details.

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 73 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 74

Review Window Manager: Motivation

Base Window System Apps
Position and decorate windows
UITK
Map n applications with virtual resources WM Provide Look&Feel for interaction with WS to 1 hardware BWS

So far: applications can output to windows Offer shared resources, synchronize access GEL

HW User control defined by application Windows & canvas, graphics contexts,
color tables, events May result in inhomogeneous user experience

Event multiplexing and demultiplexing Now: let user control windows
Window hierarchies
Independent of applications
BWS & OS: single address space, kernel
User-centered system view ext., user process
BWS provides mechanism vs. WM implements policy

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 75 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 76 Window Manager: Structure Screen Management
What is rendered where on screen? (layout question)
Where empty space? What apps iconified? (practical q's) Application-independent user interface
Example: Negotiating window position
Application requests window at (x,y) on screen; Apps Appearance Behavior ("Feel") Look & Feel ignores position afterwards by using window coordinate system UITK ("Look") WM
BWS needs to know window position at any time to handle Tiling, Pop-up menu at Techniques coordinate transformation, event routing, etc. (manages w) BWS Overlapping,... click
User wishes to move window to different position GEL Communicate HW Request position Fetch events
Or: Requested position is taken by another window change,... with BWS
Three competing instances (same for color tables,...)
Solution: Priorities, for example:
Prior (app) < Prior (WM) < Prior (user)
WM as advising instance, user has last word

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 77 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 78

Session Management Session Management

Certain tasks are needed for all apps in consistent way
WM techniques continued
Move window, start app, iconify window
Direct manipulation
Techniques WM uses for these tasks
Manipulate onscreen object with real time feedback

Menu techniques Drag & drop,...

Fixed bar+pull-down (Mac), pop-up+cascades (Motif),... Early systems included file (desktop) manager in window
manager; today separate "standard" application (Finder,...) Window borders

Icon technique: (de)iconifying app windows Created by WM, visible/hidden menus, buttons to
iconify/maximize, title bar Layout policy: tiling, overlapping
Studies show tiling WM policy leads to more time users spend rearranging windows

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 79 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 80 Session Management Session Management

WM techniques continued
WM techniques continued
Input focus: Various modes possible
Look & Feel evolves hand-in-hand with technology
Real estate mode (focus follows pointer): mouse/kbd/...
Audio I/O input goes to window under specific cursor (usually mouse)
Gesture recognition
Listener mode: input goes to activated window, even when
2.5-D windows (implemented by WM, BWS doesn't know) mouse leaves window
Transparency
Click-to-type: Special listener mode (clicking into window
To consider: activates it) - predominant mode today
Performance hit?
Virtual screens
Just beautified, or functionally improved?
Space for windows larger than visible screen
Mapping of screen into space discrete or continuous

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 81 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 82

Late Refinement Levels of Late Refinement

WM accompanies session, allows user to change
Per session, for all users window positions, etc. (changing app appearance)
System-wide information (table, config file,...) read by WM
For this, application must support late refinement
Per application, for all users
App developer provides defaults that can be changed by user
Description for each application, in system-wide area
Attributes must be publicised as configurable, with possible
Per application, per user values
Description file for each application, stored in home
App can configure itself using startup files (may be directory inconsistent), or WM can provide those values when starting
app Per application, per launch

With several competing instances: priorities Using startup parameters (options) or by specifying specific (static/dynamic!...) other description file

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 83 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 84 Implementing Late Refinement Window Manager: Location
Table files
Key-value pairs, with priority rule for competing entries
WM=client of BWS, using its access functions
Usually clear text (good idea), user versions usually editable
WM=server of apps, can change their appearance
Modern versions: XML-based
Several possible architectures Apps
WM-internal database
WM as upper part of BWS
WM Access only via special editor programs
Saves comms overhead BWS
Allows for syntax check before accepting changes, but less
But overview suffers Apps GEL transparent; needs updating when users are deleted,.....
WM as separate server WM HW

Random Rant: Why Non-Clear-Text Config Files Are Evil More comms BWS
Delta technique
But exchangeable WM GEL
Starting state + incremental changes; undo possible HW

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 85 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 86

Window Manager: Location Window Manager: Conventions

Apps
Separate user process WM
Visual consistency
Uses mechanism of shared resources
For coding graphical information across apps
BWS
E.g., requests window position from BWS, GEL Reduce learning effort checks its conformance with its layout policy, HW
Behavioral consistency and requests position change if necessary
Central actions tied to the same mouse/kbd actions (right-
More comms, but same protocol as between apps & BWS; click for context menu, Cmd-Q to quit) - predictability no direct connection app—WM
Description consistency
Syntax & semantics of configuration files / databases consistent across all levels of late refinement
Usually requires defining special language

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 87 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 88 Window Manager: Summary Administrative Details

WM leads from system- to user-centered view of WS
Class times will need to remain at TuTh 9:15-10:45
Accompanies user during session
Beginning Apr 30, classes will be held in the iRoom
Potentially exchangeable (Gates Building, basement, room B23)
Allows for implementation of new variants of desktop
Thu: UI Toolkit metaphor without having to change entire system
Next week:
E.g., still much room for user modeling (see, e.g., IUI 2002)
Jan @ CHI 2002, Minneapolis
WM requires UI Toolkit to implement same
Design sessions on final SWS assignment with David Look&Feel across applications
Afterwards, Part II begins
Craig Latta: Smalltalk/Squeak

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 89 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 90

Review: Window Manager

Implements user interface (Look&Feel policy) for interaction with windows
Window borders, menus, icons, direct Lecture 6 Apps manipulation, layout policies UITK
Virtual screens, 2.5-D,... WM
BWS Accompanies user during session, Thu, Apr 18, 2002 GEL across applications HW
Late refinement
per-system, per-app, per-user settings
Implementation
with BWS, separate server, user process

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 91 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 92 User Interface Toolkit UITK: Concept

Apps UIDS/UIDL
Interface Guidelines (Look&Feel) Two parts WM UITK
BWS Complex widgets Widget set (closely connected to WS)
GEL Elementary widgets UIDS (User Interface Design Systems, support UI design HW task

Motivation: Deliver API Assumptions
UIs decomposable into sequence of dialogs (time) using
problem/user-oriented instead of hardware/BWS- specific widgets arranged on screen (space)

50–70% of SW development go into UI All widgets are suitable for on-screen display (no post- desktop user interfaces)
UITK should increase productivity
Note: decomposition not unique

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 93 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 94

UITK: Structure UITK: Structure

Constraints
OO Toolkits
User works on several tasks in parallel -> parallel apps
Widget handles certain UI actions in its methods, without
Widgets need to be composable, and communicate with involving app other widgets
Only user input not defined for widget is passed on to app
Apps using widget set (or defining new widgets) should be asynchronously (as seen from the app developer) reusable
Matches parallel view of external control, objects have their
own "life" Structure of procedural/functional UITKs

Advantage: Subclass new widgets from existing ones Matched procedural languages and FSM-based, linear
description of app behavior Disadvantage:

But: Apps not very reusable Requires OO language (or difficult bridging, see Motif)
Debugging apps difficult

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 95 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 96 UITK: Control Flow Defining Widgets

Procedural model:
Widget:=(W=(w1..wk), G=(g1..gl), A=(a1..am), I=(i1..in))

App needs to call UITK routines with parameters Output side: windows W, graphical attributes G

Control then remains in UITK until it returns it to app Input side: actions A that react to user inputs I

Mapping inputs to actions is part of the specification, can change OO model: even at runtime
App instantiates widgets
Actions can be defined by widget or in callback
UITK then takes over, passing events to widgets in its own
Each widget type satisfied a certain UI need event loop
Input number, select item from list,...
App-specific functionality executed asynchronously in callbacks (registered with widgets upon instantiation)
Control flow also needed between widgets

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 97 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 98

Simple Widgets In-Class Exercise: Button
Elementary widgets
Universal, app-independent, for basic UI needs
What are the typical components (W,G,A,I) of a
E.g., button (trigger action by clicking), label (display text), button? menu (select 1 of n commands), scrollbar (continuous
Sample solution: display and change of value), radio button (select 1 of n
W=(text window, shadow window) attributes)
G=(size, color, font, shadow,...)
A=(enter callback, leave callback, clicked callback)
I=(triggered with mouse, triggered with key, enter, leave)

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 99 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 100 Simple Widgets Complex Widgets

Container widgets
Applications will only use subset of simple widgets
Layout and coordinate other widgets
But also have recurring need for certain widget
Specification includes list C of child widgets they manage combinations depending on app class (text editing,
Several types depending on layout strategy CAD,...)
Elementary & Container widgets are enough to create
Examples: file browser, text editing window applications and ensure look&feel on a fundamental
Two ways to create complex widgets level
Composition (combining simple widgets)
Refinement (subclassing and extending simple widgets)
Analogy in IC design: component groups vs. specialized ICs

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 101 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 102

Widget Composition Widgets and Windows

Creating dynamic widget hierarchy by hierarchically
The dynamic widget tree usually matches geographical contains organizing widgets into the UI of an application relation of associated BWS windows

Some will not be visible in the UI But: Each widget usually consists of several BWS windows

-> Each widget corresponds to a subtree of the BWS window Starting at root of dynamic widget tree, add container tree! and other widgets to build entire tree
-> Actions A of a widget apply to is entire geometric range
Active widgets usually leaves except where covered by child widgets
Dynamic because it is created at runtime
-> Graphical characteristics G of a widget are handled using
Can even change at runtime through user action (menus,...) priorities between it, its children, siblings, and parent

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 103 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 104 Refinement of Widgets Late Refinement of Widgets

Create new widget type by refining existing type
App developer can compose widgets
Refined widget has mostly the same API as base
Widget developer can refine widgets widget, but additional or changed features, and fulfils
-> User needs way to change widgets Style Guide
-> Should be implemented inside toolkit
Not offered by all toolkits, but most OO ones
Solution: Late Refinement (see WM for discussion)
Refinement creates the Static Hierarchy of widget
Late refinement cannot add or change type of widget subclasses characteristics or the dynamic hierarchy
Example: Refining text widget to support styled text
But can change values of widget characteristics (changes mostly G), or hypertext (also affects I & A)

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 105 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 106

Style Guidelines Types of UIDS
Language-oriented
How support consistent Look&Feel?
Special language (UIL) specifies composition of widgets
Document guidelines, rely on developer discipline
Compiler/interpreter implements style guidelines by
E.g., Macintosh Human Interface Guidelines (but included checking constructs commercial pressure from Apple & later user community)
Interactive
Limiting refinement and composition possible
Complex drawing programs to define look of UI
Containers control all aspects of Look&Feel
Specifying UI feel much more difficult graphically
Sacrifices flexibility
Usually via lines/graphs connecting user input (I) to actions
UIDS (A), as far as allowed by style guide
Tools to specify the dialog explicitly with computer support
Automatic
Create UI automatically from spec of app logic (research)

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 107 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 108 Assignment: Window Manager

See handout Lecture 7

Tue, Apr 30, 2002

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 109 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 110

Smalltalk & Squeak Smalltalk: Architecture

Guest lecture by Craig Latta, IBM TJ Watson
Common ancestor of all window systems Research Center
PARC, early 70's, initially on 64K Alto
See course website for his course notes, handouts, and Apps
Complete universe, simplest WS archit. UITK
system images to run Squeak on the Macs in Meyer WM OS, language, WS, tools: single address
General information about Squeak at BWS space, single process structure, communicate with procedure calls http://www.squeak.org/ GEL HW
Initially, OS & WS merged,on bare machine
Later, WS on top of OS, but still "universe"
Introduced windows, scrolling, pop-up menus, virtual desktop, MVC

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 111 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 112 Smalltalk: Evaluation Smalltalk: Evaluation
Availability: high (Squeak,...)
Adaptability: low (no explicit structured user resource
Productivity: medium (depending on tools, libs used) concept; although storing entire image possible)
Parallelism: originally none, now external
Resource sharing: high
But linguistic crash protection
Distribution: none originally, yes with Squeak
Performance: medium (high OO overhead since
everything is an object) API structure: pure OO, Smalltalk language only

Graphic model: originally RasterOp API comfort: initially low, higher with Squeak&Morphic
Style: flexible (see Morphic, for example)
Independency: High (due to MVC paradigm)
Extensibility: highest (full source available to user,
code browser) Communication: flexible (objects pass messages)

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 113 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 114

The Apple Macintosh

Introduced in 1984
Lecture 8 Based on PARC Smalltalk, Star, Tajo
Few technical innovations (QuickDraw)
Otherwise, rather steps back Thu, May 2, 2002
But landmark in UI design and consistency policies
First commercially successful GUI machine
Advertised with what is sometimes considered the best commercial in history: http://www.apple-history.com/movies/1984.mov

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 115 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 116 Macintosh: Architecture The Macintosh Toolbox

One address space, communication
Sets of routines to do various tasks with procedure calls
Functional, not object-oriented (originally written in Pascal) Apps RAM
"No" OS—app is in charge,
Organized into Managers UITK everything else is a subroutine WM Toolbox BWS in ROM library ("Toolbox") GEL (+RAM
Functional, not object-oriented from disk) HW (originally written in Pascal)
Organized into Managers
Mostly located in "the Mac ROM"

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 117 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 118

Event Manager Control Manager
Controls: Buttons, checkboxes, radio buttons, pop-up
struct EventRecord { Event loop core of any Mac app short what; // type of event menus, scroll bars,...
long message; // varies depending Processes events (from user or // on type
long when; // Timestamp in ticks Control Manager: Create, manipulate, redraw, track & system) and responds Point where; // mouse position // in global coords respond to user actions
Event Manager offers functions short modifiers; // modifier keys }; held down to deal with events
extern pascal Boolean Event types GetNextEvent(short eventMask, enum { EventRecord *theEvent); nullEvent = 0, Dialog Manager mouseDown = 1,
mouseUp = 2,
Cooperative Multitasking keyDown = 3, Create and manage dialogs and alerts
keyUp = 4, External: App must allow user autoKey = 5,
updateEvt = 6, (System-) modal, movable (application-modal), or to switch to other apps diskEvt = 7,
activateEvt = 8, modeless dialog boxes—choice depends on task! Internal: App must surrender osEvt = 15, processor to system regularly };

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 119 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 120 Window Manager(!) Finder Interface
Not the Window Manager from our layer model
Defining icons for applications and documents
Create, move, size, zoom, update windows
Interacting with the Finder
App needs to ensure background windows look deactivated (blank scrollbars,...) Other Managers
Scrap Manager for cut&paste among apps Menu Manager
Standard File Package for file dialogs
Help Manager for balloon help
Offers menu bar, pull-down, hierarch. & pop-up menus
TextEdit for editing and displaying styled text
Guidelines: any app must support Apple, File, Edit,
Help, Keyboard, and Application menus Memory Manager for the heap
List Manager, Sound Manager, Sound Input Manager,...

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 121 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 122

Resource Manager Resource Manager
Resources are basic elements of any Mac app: Descriptions of menus, dialog boxes, controls, sounds,
Identified by type (4 chars) and ID (integer) fonts, icons,...
Standard resource types (WIND, ALRT, ICON,...)
Makes it easier to update, translate apps
Custom resource types (defined by app)
Stored in resource fork of each file
Read and cached by Resource Manager upon request
Each Mac file has data & resource fork
Priorities through search order when looking for resource
Data fork keeps application-specific data (File Manager)
Last opened document, other open docs, app, system
Resource fork keeps resources in structured format
Can write resources to app or document resource fork (Resource Manager)
E.g., last window position
For documents: Preferences, icon, window position
For apps: Menus, windows, controls, icons, code(!)

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 123 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 124 ResEdit Macintosh: Evaluation
Graphical Resource
Availability: high (apps from 1984 still run today!) Editor (Apple)
Productivity: originally low (few tools except ResEdit; Mac was
Overview of resources designed for users, not programmers) in resource fork of any
Parallelism: originally none, later external+internal file (app or doc), sorted
External: Desk accessories, Switcher, MultiFinder by resource type
Internal: Multi-processor support in mid-90's

Opening a type shows Performance: high (first Mac was 68000 @ 1MHz, 128K RAM)
resources of that type sorted by their ID Graphic model: QuickDraw (RasterOp+fonts, curves...)
Style: most consistent to this day (HI Guidelines, Toolbox)
Editors for basic resource types built in (ICON,DLOG,...)
Extensibility: low (Toolbox in ROM, later extended via System
Big productivity improvement over loading resources file) as byte streams CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 125 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 126

In-Class Exercise: Macintosh: Evaluation Simple Mac Application
Adaptability: medium (System/app/doc preferences in
Write a simple Macintosh application that opens a resources, but limited ways to change look&feel) window and exits upon mouse click
Resource sharing: medium (fonts, menu bar shared by apps,...)
Distribution: none
API structure: procedural (originally Pascal)
API comfort: high (complete set of widgets)
Independency: Medium (most UI code in Toolbox)
Communication: originally limited to cut&paste

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 127 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 128 void main (void) { WindowPtr window; Rect rect;

InitGraf (&qd.thePort); // must be called before any other TB Manager (IM IX 2-36) InitFonts (); // after ig, call just to be sure (IM IX 4-51) FlushEvents(everyEvent,0); // ignore left-over (finder) events during startup InitWindows (); // must call ig & if before (IM Toolbox Essentials 4-75; IM I 280)

InitCursor (); // show arrow cursor to indicate that we are ready Lecture 9

SetRect (&rect, 100, 100, 400, 300);

window = NewCWindow (NULL, &rect, "\pMy Test", true, documentProc, Tue, May 7, 2002 (WindowPtr) -1, FALSE, 0);

do { } while (!Button());

DisposeWindow (window);

} CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 130

Review: Classic Mac OS The X Window System ("X")
Asente, Reid (Stanford): W window system for V OS,
Designed for the user, not the developer (1982)

First commercially succesful GUI system W moved BWS&GEL to remote machine, replaced local library calls with synch. communication
Technically few advances
Simplified porting to new architectures, but slow under Unix
One address space, one process, "no" OS

But revolutionary approach to UI consistency (HI Guidelines) MIT: X as improvement over W (1984)

Asynchronous calls: much-improved performance Macintosh Toolbox

Application=client, calls X Library (Xlib) which packages Pascal procedures grouped into Managers, ROM+RAM and sends GEL calls to the X Server and receiving events
Extended as technology advanced (color, multiprocessing,...), using the X Protocol. but architecture was showing its age by late 90s
Similar to Andrew, but window manager separate
Inspiration for other GUIs, esp. MS Windows
X10 first public release, X11 cross-platform redesigned

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 131 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 132 X: Architecture X Server

X is close to
X11 ISO standard, but limited since static protocol architecture
X server process combines GEL and BWS Application model
Responsible for one keyboard (one EL), but n physical Widget Set screens (GLs) WM Xt Intrinsics UITK
Xlib Xlib One machine can run several servers
Network Applications (with UITK) and WM are clients
X Server BWS+GEL GEL: Direct drawing, raster model, rectangular clipp. HW
X-Server layers: Device-dependent X (DDX), device- independent X (DIX)
BWS can optionally buffer output regions

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 133 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 134

Typical Xlib application X Protocol (pseudocode) #include Xlib.h, Xutil.h Display *d; int screen; GC gc; Window w; XEvent e;
Between X server process and X clients (incl. WM) main () {
d=XOpenDisplay(171.64.77.1:0); async, bidirectional byte stream, order guaranteed by screen=DefaultScreen(d); transport layer w=XCreateSimpleWindow(d, DefaultRootWindow(d), x,y,w,h,border,BlackPixel(d),WhitePixel(d)); // foreground & background
Implemented in TCP, but also others (DECnet,...) XMapWindow(d, w);
Creates about 20% time overhead with apps over network gc=XCreateGC(d, w, mask, attributes); // Graphics Context setup left out here
Four packet types XSelectInput(d, w, ExposureMask|ButtonPressMask); while (TRUE) {
Request, (Client->Server) XNextEvent(d, &e);
Reply, Event, Error (Server->Client) switch (e.type) {
case Expose: XDrawLine (d, w, gc, x,y,w,h); break; Packets contain opcode, length, and sequence of case ButtonPress: exit(0); resource IDs or numbers } } } CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 135 Window Manager X: Resources
Ordinary client to the BWS
Communicates with apps via hints in X Server
Logical: pixmap, window, graphic context, color map,
Look&Feel Mechanisms are separated from Look&Feel Policy visual (graphics capabilities), font, cursor
Late refinement (session, user, application, call)
Real: setup (connection), screen (several), client
Dynamically exchangeable, even during session

All resources identified via RIDs twm, ctwm, gwm, mwm (Motif), olwm (OpenLook), rtl (Tiling), ...
Implement different policies for window & icon placement,
Events: as in ref. model, from user, BWS, and apps, appearance, all without static menu bar, mostly pop-ups, flexible piped into appropriate connection listener modes

X Server is simple single-entrance server (round- No desktop functionality (separate app)
robin), user-level process Only manages windows directly on background (root) window, rest managed by applications (since they don't own root window space)

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 137 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 138

X: UITK X: UITK

X programming support consists of 3 layers
Xlib offers functions to create, delete, and modify server
resources (pixmaps, windows, graphic contexts, color maps, Xlib: visuals, fonts), but app has to do resource composition
Lowest level, implements X protocol client, procedural (C)
Display (server connection) is parameter in most calls
Programming on the level of the BWS

X Toolkit Intrinsics (Xt) Hides networking, but not X server differences (see
"Visual") Functions to implement an OO widget set class (static) hierarchy

Packages requests, usually not waiting for reply (async.) Programming library and runtime system handling widgets

At each Xlib call, checks for events from server and creates Exchangeable (InterViews/C++), but standard is in C
queue on client (access with XGetNextEvent()) Each widget defined as set of "resources" (attributes)
Extensions require changing Xlib & Xserver source & (XtNborderColor,...) protocol CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 139 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 140 X: UITK UITK

X Toolkit Intrinsics
X Toolkit Intrinsics
Just abstract meta widget classes (Simple, Container, Shell)
Xt Functions (XtRealizeWidget(),...) are generic to work
At runtime, widgets have 4 states with all widget classes

Created (data structure exists, linked into widget tree, no Parameters are generic—how? window)
-> One list (variable length) of key/value pairs, filled
Managed (Size and position have been determined—policy) (XtSetArg()), then passed into function. Order irrelevant.
Realized (window has been allocated in server; happens
Event dispatch: automatically for all children of a container)
Defined for most events in translation tables (I->A) in Xt
Mapped (rendered on screen)—may still be covered by
-> Widgets handle events alone (no event loop in app)! other window!
App logic in callback functions registered with widgets

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 141 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 142

For Thursday

Complete the Mac OS assignment by Wednesday
Have a look at the X paper on the class web site Lecture 10
We will meet in Meyer, then go over to Sweet Hall as needed, to avoid missing each other in Sweet Hall. Thu, May 9, 2002

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 143 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 144 Widget Sets What Is Motif?

Collection of user interface components
Style Guide (book) for application developer
Together with WM, define look&feel of system
Widget set (software library) implementing Style
Several different ones available for X Guide
Athena (original, simple widget set, ca. 20 widgets, 2-D, no
Window Manager (mwm) strong associated style guide) — Xaw... prefix
UIL (User Interface Language)
Motif (Open Software Foundation, commercial, 2.5-D widget set, >40 widgets, industry standard for X, comes with style guide and UIL)—Xm... prefix
Programming model already given in Intrinsics
Motif just offers convenience functions

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 145 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 146

The Motif Widget Set Programming with Motif

Simple Widgets: XmPrimitive
Initialize Intrinsics

XmLabel, XmText, XmSeparator, XmScrollbar,... Connect to server, allocate toolkit resources

Shell Widgets: Shell Create widgets
Building the dynamic widget tree for application
Widgets talking to Window Manager (root window children)
Tell Intrinsics to manage each widget
Application shells, popup shells,...
Realize widgets
Constraint Widgets: XmManager
Sensitize for input, per default also make visible (map)
Containters like XmDrawingArea, XmRowColumn,...
Register callbacks
Complex widgets like XmFileSelectionBox,...
Specify what app function to call when widgets are triggered
Event loop
Just call Intrinsics (XtMainLoop()) – app ends in some callback!

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 147 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 148 hello.c: A Simple Example #include #include #include Resource files in X #include #include
Where does the title for the PushButton come from? void ExitCB (Widget w, caddr_t client_data, XmAnyCallbackStruct *call_data)
{ -> Resource file specifies settings for application XtCloseDisplay (XtDisplay (w));
Application.PathToWidget.Attribute: Value exit (0); Syntax: }
Resource Manager reads and merges several resource void main(int argc, char *argv[]) files (system-, app- and user-specific) at startup (with { Widget toplevel, pushbutton; priorities as discussed in reference model) toplevel = XtInitialize (argv [0], "Hello", NULL, 0, &argc, argv); pushbutton = XmCreatePushButton (toplevel, "pushbutton", NULL, 0); File "Hello": XtManageChild (pushbutton); Hello.pushbutton.labelString: Hello World XtAddCallback (pushbutton, XmNactivateCallback, (void *) ExitCB, NULL); Hello.pushbutton.width: 100 Hello.pushbutton.height: 20 XtRealizeWidget (toplevel); XtMainLoop (); CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 150 }

User Interface Language UIL X/Motif: Evaluation

Resource files specify late refinement of widget
Availability: high (server portability), standard WS for attributes, but cannot add widgets Unix
Idea: specify actual widget tree of an application
Productivity: low for Xlib-based and widget outside C source code, in UIL text file development, but high using widget set, esp. Motif
C source code only contains application-specific callbacks,
and simple stub for user interface Parallelism: external yes, internal no - in original
UIL text file is translated with separate compiler design, one app can freeze server with big request
At runtime, Motif Resouce Manager reads compiled UIL file
Performance: fairly high (basic graphics were faster to construct dynamic widget tree for app than Windows on same hardware), widget sets add
Advantage: UI clearly separated from app code graphical and layout overhead, but can hold client-side
Decouples development resources

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 151 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 152 X/Motif: Evaluation X/Motif: Evaluation

Graphics model: RasterOp
API structure: Xlib procedural, Xt/widget set OO
Style: exchangeable through widget set and WM
Graphics apps need to use both APIs!
Note: apps cannot rely on a certain WM functionality
API comfort: high with Motif (even UIDS available)
Extensibility: low
Independence: low with Xlib (visuals), high with
Requires modifying Xlib source, usually also Xt and widget Motif set source, applications using extension not backwards
compatible and portable anymore Communicating apps: via RIDs in server for
Adaptability: very high (multiple resource files, UIL) resources, clipboard for text & graphics
Resource sharing: possible via RIDs
Distribution: yes, BWS, WM & apps on different

machines CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 153 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 154

Assignment

Implement a simple Motif program, such as our original ColorChooser example, or something similar that you find interesting (due Tuesday) Lecture 11
Tip: Use the interactive Xmtutor Motif tutorial application (installed for you on the Leland systems) Tue, May 14, 2002 for general information on Motif programming, and to find out more about how to use the various widget types in your application
http://www.stanford.edu/~borchers/xmtutor/

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 155 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 156 Java: AWT AWT: Architecture

AWT: Abstract Windowing Toolkit
Java is not a complete OS
Java's original window and graphics system
No OS-level Window
Even Swing is based on AWT Apps Manager UITK
Developed originally in about 6 weeks for Java 1.0 ...
Applications use the AWT for WM AWT
Each widget has its own operating-system level BWS graphical input and output GEL
window (heavyweight widgets) HW JVM The AWT works on top of the
Swing widgets do not have their own OS-level window, and Java Virtual Machine (JVM) get drawn by their container (lightweight widgets)
Swing uses AWT containers to render its toplevel widgets

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 157 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 158

Applets vs. Applications Minimum Applet: MyApplet.java

Java offers two types of GUI programs: import java.applet.Applet;
import java.awt.*; Applets public class myApplet extends Applet {
run inside a web browser using a plugin public void paint(Graphics g) {
are embedded into an HTML page g.drawString("Hello World!", 60,100);
}}
have limited access to the underlying OS (sandbox)
are subclasses of Applet Embed into HTML page:
Applications
run as standalone executables, with (almost) full OS access Alternate text

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 159 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 160 The AWT Widget Hierarchy The Component Class
Component
MenuComponent

Button MenuBar

Canvas
MenuItem Parent class for all things to see and interact with on-
Checkbox
CheckboxMenuItem
Choice
Menu screen (except for menus: MenuComponent)
Container
PopupMenu
Over 150(!) methods
Panel
ScrollPane Other interesting AWT classes:
From getWidth() to addMouseMotionListener()

Window MenuShortcut

Dialog
Event See (general pointer for all documentation):
FileDialog
EventQueue
Frame
Font http://java.sun.com/j2se/1.3/docs/api/java/awt/Component.html
Label
FlowLayout, CardLayout... (We are using 1.3.1)
List
Scrollbar
TextComponent
TextArea
TextField

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 161 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 162

Event Model Event Model

EventListeners receive event of corresponding type
Original model: action() method of toplevel widget
ActionEvent, MouseEvent, WindowEvent,... import java.awt.*; (e.g., Applet) would receive and handle all events public class Hello extends Frame implements ActionListener { public static void main(String argv[])
Became huge method for large apps; inefficient { new Hello();
Since Java 1.1: Delegated event handling }
public Hello() { Must register for events to receive them, by implementing an Button myButton = new Button("Hello World"); event listener interface myButton.addActionListener(this);
add(myButton, "Center"); Examples: ActionListener (for button clicks), setSize(200, 100); MouseListener, MouseMotionListener, WindowListener setLocation(200,200); setVisible(true); (when window is activated, iconified,...), etc. }
public void actionPerformed(ActionEvent e) { Smaller, dedicated listener methods, better performance exit(0); } } CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 163 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 164 Overview of Swing Components Lecture 12 (David)

Thu, May 18, 2002

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 165 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 166

Some Swing Examples (David) Java’s Evolution

Toolbar sample app
Java’s evolution with respect to its GUI and media-
Tabbed pane sample app handling features looks like a fast-forward replay of the history of window systems:
Table sample app
Java 1.0 (1995): 6-week version of AWT
Java 1.1: Delegated event model, localization
Java 2, v.1.2: JFC (Swing, Java2D, Accessibility, Drag&Drop), audio playback
Java 2, v.1.3: audio in, MIDI, Timer (for UI, animations, etc.)
Java 2, v.1.4 (2002): full-screen mode, scrollwheels, Preferences API

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 167 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 168 Java: Evaluation Java: Evaluation
Graphics model: RasterOp
Java2D offers vector model, but it is not used by the WS
Availability: high (binary portability), better for AWT
Style: native like the OS (AWT), pluggable-simulated (Swing)

Productivity: medium with AWT, high with Swing Note: Window Manager is supplied by the OS
Extensibility: medium
Parallelism: external yes, internal depends on OS
New widgets can be subclassed easily, but adding features to the
underlying WS is not intended (e.g., no support for input devices other Performance: medium (bytecode interpretation of than mouse and keyboard) class files), memory and perfomance tradeoffs
Adaptability: fairly high (with Swing)
between AWT (native widgets) and Swing (simulated Developers can implement new look&feel, and switch it at runtime
ResourceBundles can store resources (typically, texts and icons for widgets) different languages), similar to Mac OS resource forks
But: geared towards localization; no combination of multiple files; no user access intended (not a clear-text format)
Resource sharing: depends on core OS
Distribution: no (nowadays, distributing objects is more common)

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 169 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 170

Java: Evaluation

API structure: purely OO
API comfort: high, esp. with Swing (3rd-party UIDSs available) Lecture 13
Independence: high (class concept), esp. with Swing (supports MVC) Tue, May 21, 2002
Communicating apps: clipboard for text & graphics, drag&drop (from Java apps to and from other Java and native apps),

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 171 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 172 Mac OS X Mac OS X: Architecture

OS: Unix Apps
Unix Mach microkernel (Darwin, open source) UITK Cocoa (, Carbon, JDK, Classic)
-> Protected memory, preemptive multitasking WM Finder (user-level process)
GEL Quartz Core Graphics Rendering (vector), OpenGL, QuickTime -> Single application cannot corrupt/freeze entire system !
BWS Quartz Core Graphics Services (pixmap) Graphics library (“Quartz”): Display PDF HW
Roots: NeWS (Display PostScript)
Vector-based
UITK: Cocoa
OO framework
Written in Objective-C, but interfaces to Java, C, and C++
Implements Aqua Look&Feel Apple’s Layer Model of Mac OS X

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 173 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 174

Apple’s Layer Model of Mac OS X Event Handling

Similar to our Reference Model
Window Server distributes events to per-application
Cocoa is the “native” API (can be used with Obj-C or Java) (really per-process)
JDK is used for 100% Java/Swing applications queues
Carbon is an updated version of the old Macintosh Toolbox
Used to easily port existing applications, Carbon apps run on 9&X
Classic emulates Mac OS 9 to run old apps unmodified
BSD is used to run existing standard Unix applications
-> Mac OS X tries to please everyone (at the expense of high cost for supporting multiple APIs)

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 175 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 176 Dynamic Typing, Binding, and Objective-C Linking
Implementation language of the Cocoa framework
C++ is a static language, Java and Obj-C are dynamic
Created in 1983 to combine OO principles with C
C++: Cannot use a superclass (eg TObject) as a subclass (eg
In its concepts very similar to Java, unlike C++ TShape) even though you know you could; the compiler
Dynamic typing, binding, and linking prevents such code from building
Introduces new constructs to C

[object message:par1 par2:type2] is analogous to Java’s Obj-C & Java move this check to run-time object.method(par1, par2)
In C++, a superclass must either contain all the methods any
- for instance methods, + for class methods subclass will use, or it must be mixed in using multiple
id corresponds to void *, self corresponds to this inheritance. To add a method to the superclass, all subclasses
@ compiler directives (@interface..@end, @implementation..@end,…) must be recompiled (fragile bass class problem)
Protocols are analogous to Java’s interfaces
Dynamic Binding avoids bloated superclasses and minimizes
Classes are objects of type Class recompilation due to this problem

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 177 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 178

Cocoa Cocoa Class Hierarchy
The UITK of Mac OS X
Evolved out of NeXTSTEP (which was released in 1987–just four NSObject
Fairly flat hierarchy years after the Macintosh was introduced–, and which later became NSEvent
Reason: Delegates and OPENSTEP) NSResponder protocols can be used to mix
Cocoa’s class names still give away its heritage (NSObject,…) NSWindow in functionality, no deep
Two main parts NSView NSControl
subclassing required Foundation: Basic programming support NSButton etc.
NSObject, values, strings, collections, OS services, notifications, NSApplication archiving, Obj-C language services, scripting, distributed objects NSCell (lightweight controls)
AppKit: Aqua interface NSMenu
Interface, fonts, graphics, color, documents, printing, OS support, NSMenuItem international support, InterfaceBuilder support etc.
Largest part (over 100) are interface widgets
Complex hierarchy, see Online Help in Project Builder

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 179 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 180 Responders, Events, Actions, Responders, Events, Actions, Outlets Outlets
Events generated by user or system (eg periodic events)
In each step of the responder chain, a delegate can be
Actions are generated by objects (eg menus) in response to lower-level events given a chance to handle the event

InterfaceBuilder lets developer connect actions to custom objects (e.g., Applications, windows, views etc. can be extended by from a button to a custom class), using “IBAction” constant in the source adding a delegate without having to subclass them
Most objects are subclasses of NSResponders and can respond to
Outlets are instance variables to refer to other objects events

In static frameworks (e.g., those based on C++), each object needs large InterfaceBuilder knows about them and lets the developer switch statement to determine if it can handle an event connect outlets graphically (“IBOutlet” constant)
In Cocoa (dynamic framework), NSApplication can find a responder that
Example: A custom class that wants to display a result in a can handle an event (respondsToSelector), then call its method directly text field needs an outlet
Framework takes care of Responder Chain
Events are passed on along the responder chain (key window ¥ main window ¥ application) until they can be handled by some object

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 181 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 182

Interface Builder Interface Builder: Example
Graphical tool to create user interfaces for Cocoa applications
Allows developer to not just visually define the widgets in a UI (i.e., specify the static layout of the user interface–which is what most UIDS support), but also define the connections between widgets and custom classes of the application that is being written (i.e., the dynamic behavior of the user interface)
UI can be tested inside IB without compiling or writing any code
Tied into development environment (Project Builder)
Suggests a more user-centered implementation process that starts with the user interface, not the application functionality
IB generates source code skeleton that can then be filled in
IB uses special constants to include hints about outlets and actions in the source code
Resources are stored in nib files (NeXTSTEP Interface Builder)
An application reads its main nib file automatically when it starts up The user input in an NSTextField is connected to the convert()
Additional nib files can be read when needed (for transient dialogs, etc.) method of a custom TempController class in an application.

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 183 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 184 Mac OS X: Evaluation Mac OS X: Evaluation

Style: native (Aqua)
Availability: medium (only on Apple hardware)
Computationally expensive (less so when using OpenGL, see above)

Productivity: very high, but learning curve (Cocoa framework) Extensibility: fairly high (due to dynamism)
New widgets can be subclassed, delegates can often help avoid
Parallelism: external yes, internal yes subclassing
Performance:
Adaptability: fairly high

High with Obj-C plists store application settings in clear-text XML files (similar to XML)

Applications are bundles (directories) of binary code and resources (each Medium-high with Java (uses Java Bridge) resource can be a Unix file)
Graphics model: Vector
Resource sharing: yes
Latest version is moving to an all-OpenGL rendering engine
Distribution: no

-> Transparency etc. done in hardware, for desktop and all Distributing objects is more common
applications Cocoa: serialization and Connection mechanisms

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 185 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 186

Mac OS X: Evaluation Assignment

API structure: OO with dynamic binding
Work through the Java Temperature Converter
API comfort: high tutorial
Interface Builder UIDS enables developer to specify not only static
Local copy on Mac OS X machines: widget layout, but also dynamic app structure /Developer/Documentation/Essentials/devessentials.html

Independence: high (class concept, MVC) On the web: http://developer.apple.com/techpubs/macosx/Cocoa/
Communicating apps: JavaTutorial/javatutorial.pdf
Pastboard for fonts, rulers, text, drag&drop data, other objects:
Just work through chapters 1 and 2; you will complete a mostly handled automatically by the framework simple Cocoa Java application, and will use Interface
Services to export functionality to other apps (“Mail text”) Builder to connect the interface to your code.
Complete by Thursday, then submit the final product to our website after David is back.

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 187 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 188 The Post-Desktop Interface

Enhanced Desktop
Add voice recognition, video, audio, etc. Lecture 14
Different Devices
PDAs, Cell Phones, Wearable PCs
Thu, May 23, 2002 Different Modalities
(Guest Lecture: Brad Johanson) Voice, tactile feedback, etc.
Multi-Device Interfaces
Ubiquitous computing, smart spaces, interactive workspaces

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 189 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 190

Principles for Ubiquitous Ubiquitous Computing Computing
Originated with Mark Weiser, “The Computer for the
Boundary Principle st 21 Century,” 1991.
Volatility Principle
Main Concept:
Semantic Rubicon
Computing technology will become part of the background.
We will be able to maintain only a peripheral awareness
Has come to also mean:
Any sort of multi-device interactions that integrate with daily life
Also called “Pervasive Computing.”

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 191 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 192 Room-based Ubiquitous Computing Some Characteristics

Basic Idea, one room for team collaboration with:
Human Constraints
Multiple devices, some permanent, some transient, some
Bounded Environment mobile
Human Centered Interaction
Multiple users collaborating
Technological Constraints
Multiple applications
Heterogeneity
Changing environment

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 193 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 194

Approaches iROS

Room OS: Make all devices look like one complex
A set of middleware infrastructure pieces virtual machine (i-Land, Darmstadt and Gaia OS,
Facilitates construction of apps for interactive UIUC)
workspaces Intelligent Environment: Create an environment that
anticipates the users needs and does the right thing. Facilitates recombining applications and devices in (Intelligent Room, MIT, Easy Living, Microsoft new ways. Research)
Meta Operating System: Allow pre-existing programs, and develop new programs using devices own toolkits. (Interactive Workspaces, Stanford)

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 195 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 196 iROS Video iROS: Discovered Principles

Moving Data
Moving Control
Dynamic Application Coordination

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 197 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 198

iROS Component Structure The Event Heap

A digital environment to mirror the physical one Interactive Workspace Applications
Obey the boundary principle iCrafter
Give applications the means to: State Service Service Data
Manager Invocation Discovery Heap Notify other apps in the environment about changes in their Persistent Store Event Heap File Stores Other APIs state
Understand what other applications are doing. Key: Stanford iROS Application Developers Other Infrastructure
React to occurrences in the environment

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 199 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 200 How Things Work Event Heap Example String EventType “ViewChange” StringStringString EventType EventTypeEventType “ViewChange” “ViewChange”“ViewChange” String TargetDevice *“FrontScreen” StringStringString TargetDevice TargetDeviceTargetDevice “FrontScreen” *“FrontScreen” int ViewNumber 137 Intintint ViewNumber ViewNumberViewNumber * 713 Controller

PutEvent waitForEvent String EventType “ViewChange” String TargetDevice *“FrontScreen” getEvent Int ViewNumber 7*

String EventType “ViewChange” StringString EventType TargetDevice “ViewChange” “FrontScreen” waitForEvent Stringint ViewNumber TargetDevice 13 * int ViewNumber 7 StringString EventType EventType “ViewChange” “ViewChange”

Viewer 1 Viewer 2 StringString TargetDevice TargetDevice “SideScreen” * intint ViewNumber ViewNumber * 7 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 201 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 202

Event Heap: System Properties Event Heap: Design Choices

Extensible
Routing Patterns
Expressive
Opaque Communication and Data Format
Simple and Portable Client API
Logically Centralized
Easy to Debug
Simple, General API
Perceptual Instantaneity
Scalable to workspace sized traffic load
Failure tolerance
Application portability

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 203 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 204 Event Heap: Important Capabilities iCrafter

Intermediation
Service Description
Snooping
Devices
Routing
Interface Generation
By source
Interface database
By application
Room configuration database
By device
By person
By group

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 205 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 206

iCrafter: Example iROS: Evaluation

Availability: high (Windows, Unix, Mac OS, easy portability)
Productivity: high, but at the level of gluing large components
Parallelism: by nature
Performance:
Medium (compared to direct socket connections)
Graphics model: Underlying UI for platform, or via iCrafter

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 207 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 208 iROS: Evaluation iROS: Evaluation
Style: based on device

API structure: mostly Java-based, but other APIs possible Extensibility: very high due to flexible typing, snooping,
intermediation, and self description eheap: event creation, manipulation, placement and retrieval
API comfort: simple, easy to learn, low level but powerful
Adaptability: high (linking apps), dep on OSs (end user
add a few lines to Java app to make it talk to the eheap config)
Independence: high– designed to keep applications independent
Resource sharing: not inherent
suggests different way of structuring UI code in an app
Distribution: inherently distributed (via central server)
Communicating apps: via Event Heap
The main feature!

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 209 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 210

Event Heap Example App: speaktext Event Heap Example App: speaker import iwork.eheap2.*; import iwork.eheap2.*; import javax.speech.*; import javax.speech.synthesis.*; class speaktext { // Connects to event heap in static void main(String []args) // arg[0], and sends an AudioEvent class speaker { // Speaks text received as AudioEvent from eheap static void main(String []args) { { // with the text in arg[1]. try { EventHeap theHeap=new EventHeap(args[0]); // Connect to the Event Heap try{ Event myEvent=new Event("AudioEvent"); // Create the template event EventHeap theHeap=new EventHeap(args[0]); // Connect to the Event Heap myEvent.setTemplateValue("AudioCommand", "Read"); // Capitalization does matter while(true) { // Loop forever retrieving Event myEvent=new Event("AudioEvent"); // Create an event try{ myEvent.setPostValue("AudioCommand", "Read"); // Set its fields Event retEvent=theHeap.waitForEvent(myEvent); // Block thread until a matching event arrives // You can also use this with a timeout, myEvent.setPostValue("Text", args[1]); // or use GetEvent which returns immediately simpleSpeak((String)(retEvent.getPostValue("Text"))); // Get text to speak out of event and say it theHeap.putEvent(myEvent); // Put event into the Event Heap } } catch(Exception e) { // Catch malformed events to keep looping e.printStackTrace(); catch(Exception e) { } e.printStackTrace(); } // End of loop getting events } } catch(Exception e) { } e.printStackTrace(); } } } // End of main()

private static void simpleSpeak(String phrase) { // Uses Java Speech library to speak text phrase; SynthesizerModeDesc mode = new SynthesizerModeDesc(); // has nothing to do with the Event Heap Synthesizer synth = Central.createSynthesizer(mode); synth.allocate(); // Get ready to speak synth.resume(); synth.speakPlainText(phrase, null); // Speak the phrase synth.waitEngineState(Synthesizer.QUEUE_EMPTY); // Wait until speaking is done } } Review

iROS: iRoom Operating System
Meta-OS to glue together apps in an interactive room Lecture 15
Implements Event Heap (similar to event queue on a single machine); simple Java API
But: more robust against failure (important in rooms) Tue, May 28, 2002
Events expire, services continuously beacon their (Guest: Merrie Ringel) availability
Mostly infrastructure-oriented, but important implications for HCI
See iStuff (today) for an example

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 213 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 214

iStuff iStuff taxonomy

Interactive Stuff: a physical UI toolkit
Similar to Card's Design Space of Input Devices
Motivation:
One vs. many bits vs. discrete
Make prototyping physical post-desktop interfaces (for
But: Additional dimensions ubiquitous computing) as simple as prototyping GUIs
Input vs. output devices
Solution:
Modality (force, heat, light, sound,...)
Build on Event Heap (assume rich infrastructure)
See
Create lightweight physical standard UI components http://www.stanford.edu/~merrie/istuff/device_table.html
buttons, lights, sliders, speakers, buzzers, ...
Combine with peer functionality in a PC to create iStuff device

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 215 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 216 iStuff Layer Model Accessing iStuff (Example: iButton)
From Java code actual wireless
Any eheap app can use iStuff without further libraries iButton
RF transmitter Sending and receiving events between apps and iStuff Conceptual
Main challenge: Abstraction and paradigm shift Parallel port driver iStuff "device"
Layer model (similar to WS reference model) Application PC daemon Event Heap Library Event Heap Library
From device-specific, to generic, to application-specific device semantics (example: date slider) Event Heap Server
iStuff-savvy applications need to receive input from multiple sources (one solution: receive all input just from eheap), and deal with multiple concurrent input (from one or more users)

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 217 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 218

Examples Examples

iSpeaker
PC with daemon waiting for speaker events
Plays audio file (in URL field of event) to sound card
Line-out connected to small standard radio transmitter
iSpeaker is a simple portable radio!
iButton
Simple extension: New event type to speak text, ASCII text
RF garage door opener style button read by PC text-to-speech software, and played back as above
RF receiver with parallel port interface
PC daemon creates eheap events from button presses
Other apps can listen for button events with specific IDs

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 219 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 220 iStuff: Evaluation
Availability: high (blueprints available, cross-platform)
Productivity: higher than building custom physical UIs
Performance: low (eheap and JVM delays)
Style: physical, custom designs possible Lecture 16
Extensibility: high (design your own hardware, or use off- the-shelf components, to add new devices)
API structure: eheap (Java as default) Thu, May 30, 2002
API comfort: easy to learn, but still similar to "first- generation" event handling models of window systems
Independence: medium (need "patch panel" abstractions)
More information: http://www.stanford.edu/~borchers/istuff/

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 221 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 222

Review: iStuff The DIA Cycle Design
Toolkit of physical user interface components
Make prototyping physical UIs as convenient as prototyping GUIs

Analyze/ Prototype/ Test/ Implement Evaluate

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 223 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 224 DIA Iterations UIDS/UIMS

What changes with each iteration?
User Interface Design Systems  Higher fidelity of prototype
Create UI visually or in language  Finer granularity of user feedback
Look (static layout)
User Interface Management Systems
Specify run-time behavior as well as visual layout
Adds specification of Feel (dynamic behavior)
Boundaries are blurred

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 225 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 226

Prototyping Tools: Paper-Based Prototyping Tools: Graphics apps

 Scenario, storyboard, paper prototype, Post-It prototype
Photoshop & Co.  Crudest forms, used in first DIA cycles
Allow for visual detail and quality  Provide best initial feedback
Easy to reuse and change  Problem: Hard to reuse or adapt to feedback (throw-away)
Drawings can become part of actual UI
More useful for non-standard GUIs
Easy to distribute electronically
Simple interactivity with help of an operator
Enable/disable layers,...
Danger of looking too polished Gayle Curtis
Limits feedback, suggests the interface is "done"

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 227 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 228 Prototyping Tools: Prototyping Tools: Animation apps
Director, Flash, LiveMotion & Co. Presentation apps
Usually implement timeline metaphor
PowerPoint & Co.
Good for intricate animations
Pixel-based (Director): Maximum control over appearance
More potential for interactivity
Vector-based (Flash, LiveMotion): Smaller files, editable objects
Timing
Powerful when extended with scripts
Animation
But: Scripting languages are clumsy by CS standards
Simple Controls
May allow for integration of non-standard hardware and other
Can be used for pitching, or as clickthrough prototype OS features (Director Xtras,...)
Easy to change, and distribute (if standard application)
Can even become final product

Good for non-standard UIs Virtual Vienna, Flash web content,...

No programming skills needed Distribution usually fairly easy (free player apps)
But: Large designs become hard to manage CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 229 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 230

Prototyping Tools: Web JavaScript Example

DHTML=HTML+JavaScript etc.
Viewable in any browser (in theory...), over the net
But: No precise look&feel (nature of the web)
What's the output? All done.

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 231 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 232 Prototyping Tools: Rapid Development Environments Prototyping Tools: Special-Purpose
VisualBasic, Delphi, RealBasic, Tcl/Tk etc.

Good for standard GUIs (create standard look&feel) Example: MAX
Often become final product
Multimedia development
Partly interpreted environment

Quick development cycle, but potential performance issues Originally for MIDI applications

Distribution: OK Extended to include graphics, audio, and video

Not always cross-platform Build applications by connecting "patchers" that process
May require specific runtime environment incoming data

RealBasic at WWDC 2002: Very helpful for specific type of applications (e.g., MIDI
"40% of tools on VersionTracker are done in RealBasic" processing, such as interactive music systems)

(But then, 40% of tools on VersionTracker s**k...) Can be used for end products (WorldBeat)

"Programming for the rest of us" (empowers users) Distribution: Free player, but platform-bound CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 233 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 234

Tcl/Tk In Depth (David) Summary: Prototyping Tools

See also David's materials online
Spectrum of tools
Homepage for everything Tcl/Tk:
From low- to high-fidelity in graphics (look) http://www.scriptics.com/
From low- to high-fidelity and complexity in interactive behavior (feel)
Tcl/Tk assignment: See online handout
From platform-specific to cross-platform
Choice depends on design stage, target platform, application type, and familiarity/learning curve
Winograd's Law: Use the tool that someone you know is familiar with :)

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 235 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 236 Tcl/Tk: Complex Interface Example

Lecture 17

Tue, June 4, 2002

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 237 CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 238

The End

Course Review
Q&A
Feedback
Finals
Written short exam on Wed Jun 12
Topics are only what's covered in this slide set

Thanks for participating!

CS377A: A Programming Approach to HCI • Jan Borchers • Spring 2002 239