WIMP interfaces

• WIMP = Windows, Icons, Menus, Pointers – they’re everywhere! – in fact, after circa 1983, the vast majority of Lecture 12: interfaces are based on the WIMP paradigm Beyond WIMP • e.g., think about Windows vs. Macintosh vs. UNIX-based window managers – some exceptions... • e.g., text terminals, game systems – ... but generally, WIMPs rule!

CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 1 CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 2

Evolution of computers (& users) Beyond WIMP

Hardware Operating Programming Terminal Advertising Generation User types User interface paradigm technology mode languages technology image • WIMP has been around a while, and probably One user at a TTY, Vacuum tubes, huge 1 time "owns" Machine typewriter. will remain on the scene for a long time. machines, much Experts, Computer as 1945-1955 machine (but language Only used in Programming cooling, short mean pioneers calculator Pioneer for a limited 001100111101 the computer time between failures. time only) center. • But let’s think forward. Batch ("computer as Transistors; more What’s the next step? 2 temple" - Line-oriented Technocrats, Computer as reliable. Computers Assembler ADD 1955-1965 make terminals professional information Command languages start seeing use A,B Historical offerings to ("glass-TTY") computerists processor outside the lab. – Nielsen: “virtual realities, head-mounted displays, sound get oracle replies) and speech, pen and gesture recognition, animation and Full screen Specialized Integrated circuits. Timesharing terminals, groups without multimedia, limited artificial intelligence, and highly 3 Businesses can cost- (online "High-level" Mechanization alphanumeric computer Full-screen strictly hierarchical 1965-1980 justify buying transaction languages, of white-collar portable computers with cellular or other wireless characters only. knowledge menus and form fill-in Traditional computers for many processing Fortran, Pascal labor Remote access (e.g. bank needs. systems) common. tellers) communication capabilities” (?!) Graphical Problem Personal 4 VLSI. Individuals can Single user displays with Business oriented productivity WIMP (Windows, Icons, – increasing computing power makes this possible 1980-1995 buy their own personal fair resolution. professionals, languages, (computer as Menus, and a Pointing device) Modern personal computer computers Desktops and hobbyists spreadsheets tool) heavy portables. – but can all this be in a single interface? "Dynabook" Wafer-scale Networked [61], maybe not... that’s yet another difference! 5 integration, computer- single user Non-imperative, multimedia I/O, Computer as Noncommand interfaces 1996-? on-a-chip. Individuals systems and possibly Everybody easily portable, entertainment • interfaces may become more task/user specific Future can buy many embedded graphical with cellular computers. systems modem.

CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 3 CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 4

Page 1 Functional vs. Object-Oriented One step forward...

• “Old-school” interfaces are functional • Thought exercise in requiring specification of entire function – imagine you have access to a movie database – uses a “verb-noun” syntax • database = info about films, actor/actresses, etc. – e.g., “rm foo”, “emacs file.java” – how can you answer questions like... • Current GUIs are object-oriented • In what films did Harrison Ford star between 1980-90? – uses a “noun-verb” syntax • In what films did Julia Roberts and Richard Gere costar? – e.g., select icon, drag to trash / select “Open” • What are the most popular Sci Fi movies of 1993? • What seems to be coming on the horizon? – “syntax-free” interfaces – well, at least syntax-flexible • can specify noun-verb, or verb-noun, or something radically different... just like communication w/ people

CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 5 CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 6

One step forward... Non-command interfaces

• Ahlberg & Shneiderman’s FilmFinder • Idea: user and computer interact not through a dialogue of commands and responses, but through a complex, dynamic, continuous interaction • Huh? What’s that? • Nielsen’s 12 “interaction characteristics” for next-generation, non-command software – not all applications will incorporate all 12 – rather, it is expected that many applications will incorporate a significant subset of the 12

CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 7 CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 8

Page 2 1. User focus 2. Computer’s role

• Interaction feels like “using a computer”, • Old: “Do what you’re told” not working on a task (according to Nielsen) New: “Do what I mean!” • With NC interfaces, focus = task... • Great idea... but why is this hard? features come for free implicitly • Example: Intelligent tutoring • Example: – monitor what the student knows Portholes – interrupt with instruction when necessary system • Example: Model tracing / “Mind tracking” – update – infer student knowledge, or disabled user’s every 5min intentions, or driver’s intentions... – implicit awareness

CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 9 CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 10

3. Interface control 4. Syntax

• Old: User controls computer • Old: Rigid interaction “syntax” New: Computer controls interaction New: No / little syntax • Examples: warn user of incoming email, • Example: deleting files infer current writing task and provide – way #1: select and delete (noun-verb) template, etc. – way #2: say “remove all *.java files” (verb-noun) • BUT this is very hard to do well – can we integrate multiple methods? – must avoid interrupting the user • Example: writing math expressions – guesses / inferences had better be right! – try not to require top-down or bottom-up

CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 11 CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 12

Page 3 5. Object visibility 6. Interaction stream

• Old-school interfaces with “direct • Old: Single-threaded input / output manipulation” require visible objects New: Multi-threaded, multimodal • New interfaces could manipulate objects • Example: “Put that there!” implicitly through higher-level interactions, – point to display wall at object or with hidden agents – say “Put that”... point to destination... “there” • Might this be dangerous? • Example: eye-driven window interface – user doesn’t know about manipulation File Edit Special – can be good, can be bad Birds Zoo Zoo

hawk dove Birds Fish

Done Trash

CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 13 CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 14

7. Bandwidth 8. Tracking feedback

• Old: Low input bandwidth (keys, mouse) • Old: Feedback only after completed input New: Very high bandwidth New: Continuous feedback on-the-fly • Systems may incorporate motion tracking, • Example: Emacs search (sort of) virtual reality, speech, “peripheral” input • Example: movie database • Difficulties • Interface should react like the real world – requires lots of processing power, both for – again, input-output lag is an issue accepting input and interpreting it – again, processing power is an issue – lags are unacceptable! (e.g., motion sickness in virtual environments)

CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 15 CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 16

Page 4 9. Turn-taking 10. Interface locus

• Old: First user, then computer, then user... • Old: Computer on the desk New: Continuous stream for both New: Computers everywhere • Closely related to feedback... computer is • Ubiquitous computing from... always responding, so there’s never a “turn” – smaller, lighter “computers” • How does this map onto the real world? • e.g., PDAs, calculators, watches – when we interact with the inanimate world? – computers built into everyday objects • e.g., ovens, cars, shopping carts • e.g., walking through the park, playing basketball – when we interact with the animate world? – computers built into not-so-everyday objects • e.g., pet dog robots • e.g., talking to someone, interacting with a pet – we sometimes take turns in the real world... • Good for awareness, “telepresence”; why not in a user interface? dangerous for privacy?

CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 17 CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 18

11. User programming 12. Software packaging

• That is, programming for end users • Old: Application-centered approach • Old: (Usually) hard-core macro languages New: System-wide, OO approach New: Smooth adaptation of objects • Example: spell checkers • Example: object-oriented customization – should be only one for your entire system – “take your basic”