How we Got Here and Where to Take it Scott Hudson HCI Institute Carnegie Mellon Instagram: Two guys, two years à 30 million users (now 300 million) What made that possible? Enablers “Making it easy to make things easy” àAdvances in UI Software Starting with tools… Starting in the early and mid-80’s, technical HCI community had a very clear goal: Make it easy enough to create these “new graphical user interface thingees” to allow people with minimal programming skills to do it (quickly & well) At the time it took days of work by programmers with fairly rare skills to build even simple interfaces Starting in the early and mid-80’s, technical HCI community had a very clear goal: Make it easy enough to create these “new graphical user interface thingees” to allow people with minimal programming skills to do it (quickly & well) At the time it took days of work by programmers with fairly rare skills to build even simple interfaces A Great Concept: User Interface Management System (UIMS) Analogy to DBMS Major line of work: Layered modular abstractions like other big systems like e.g., compilers A Great Concept: User Interface Management System (UIMS) Analogy to DBMS for (i=0; i<9; i++) { Major line of work: Layered modular if (i== 42) { abstractions like other big systems Lexer like e.g., compilers Parser Optimizer (optional) AST Once you work out the task Code Generator abstractions can refine each layer, etc… 0100101010 10101110 Lots of Systems Were Produced With This Mindset TIGER [Kasik ‘82] Lots of Systems Were Produced With This Mindset TIGER [Kasik ‘82] Syngraph [Olsen ‘83] Lots of Systems Were Produced With This Mindset TIGER [Kasik ‘82] Syngraph [Olsen ‘83] IPDA [Olsen ‘84] Lots of Systems Were Produced With This Mindset TIGER [Kasik ‘82] Syngraph [Olsen ‘83] IPDA [Olsen ‘84] ADM [Schulert ‘85] Lots of Systems Were Produced With This Mindset TIGER [Kasik ‘82] Syngraph [Olsen ‘83] IPDA [Olsen ‘84] ADM [Schulert ‘85] GWUIMS [Sibert ‘86] Lots of Systems Were Produced With This Mindset TIGER [Kasik ‘82] Syngraph [Olsen ‘83] AIDE [Hix ‘86] IPDA [Olsen ‘84] ADM [Schulert ‘85] GWUIMS [Sibert ‘86] Lots of Systems Were Produced With This Mindset TIGER [Kasik ‘82] Syngraph [Olsen ‘83] AIDE [Hix ‘86] IPDA [Olsen ‘84] ADM [Schulert ‘85] GWUIMS [SibertA ‘L86]GAE [Bergeron ‘87] Modules and Layers Boil Down to Something Like This Input Application UI Core Appl User Output Interface And this led directly to… Very little we see today… at least not directly, it took a lot more progress first (Resulting interfaces were typically mediocre) Because we had missed something really fundamental Our CS intuition/methodology told us that if we could understand and decompose “the input/processing/output problem” of UI we’d have it But we missed that this little bit on the left (“the user”) actually changes everything User User Centered Systems Becomes a much harder problem… e.g., can’t open the user’s head and pour in the right mental model To do well you have to structure each system around the user’s concepts (AND do all the rest) à The user profoundly impacted what should have been purely technical issues In the end the big advance wasn’t a particular architecture, algorithm, or system It was really a change in mind set à To be user centered even in what seemed like strictly technical issues Led to adopting interdisciplinary methodologies (and a whole new set of issues to be addressed) At the Same Time… Lots of other research was going on before, during, and after UIMS work The shift to a user centered mindset helped all of it à We now see the fingerprints of the work all over modern interactive systems UIMS Work in the Overall UI SmallTalk Research/Product Flow Java Industrial Toolkits ••• [Sun, Apple, …] Swing Early Academic Toolkits Adv. Academic Toolkits Many [CMU Andrew, MIT X Windows, …] [Stanford, CMU, UofAZ, GATech, …] Current Toolkits Input Academic UIMS Work Models Interactive systems infrastructures today are converging and we can trace many things we see there back through research toolkits that followed UIMS work Examples Three major research toolkit efforts through the 90s Stanford [Linton et al.]: Interviews and Fresco CMU [Myers et al. ]: Garnet and Amulet UofAz/GATech [Hudson et al.]: Artkit and subArctic Some traceable contributions : Concurrency model [subArctic] à Swing à many Layout abstractions [Interviews] à Swing à many Layout w/ constraints [Garnet, subArctic, Others*] à Adobe, Apple Input dispatch model (heavily modified/adapted) [Garnet/Amulet, Artkit/subArctic] à Swing à many Resizable rich icons [subArctic] à Apple, Android … Animation abstractions [Artkit, Amulet] à Android, … Examples Three major research toolkit efforts through the 90s Stanford [Linton et al.]: Interviews and Fresco CMU [Myers et al. ]: Garnet and Amulet UofAz/GATech [Hudson et al.]: Artkit and subArctic Some traceable contributions : Concurrency model [subArctic] à Swing à many Layout abstractions [Interviews] à Swing à many Layout w/ constraints [Garnet, subArctic, Others*] à Adobe, Apple Input dispatch model (heavily modified/adapted) [Garnet/Amulet, Artkit/subArctic] à Swing à many Resizable (9-part) icons [subArctic] à Apple, Android … Animation abstractions [Artkit, Amulet] à Android, … *esp. Alan Borning’s work at UWash Other Tool Successes: UI “Builders” Tools to let you draw the (graphical parts of the) interface SOS (INRIA) [Hullot ‘81] Trillium (PARC) à NeXTInterface Builder [Henderson ‘81, ’86] Principle: Visual things should be expressed visually (with minimal code) This turned out to be extremely important in “Making it easy to make things easy” à Nearly every modern IDE now supports this Microsoft produced a whole series of products: Visual Basic, Visual C++, … Visual Studio where the “Visual” part was a UI Builder Principle: Visual things should be expressed visually (with minimal code) This turned out to be extremely important in “Making it easy to make things easy” à Nearly every modern IDE now supports this Microsoft produced a whole series of products: Visual Basic, Visual C++, … Visual Studio where the “Visual” part was a UI Builder Principle: Visual things should be expressed visually (with minimal code) This turned out to be extremely important in “Making it easy to make things easy” Microsoft à Nearly every modern IDE now supports this Microsoft produced a whole series of products: Visual Basic, Visual C++, … Visual Studio where the “Visual” part was a UI Builder Principle: Visual things should be expressed visually (with minimal code) This turned out to be extremely important in “Making it easy to make things easy” Microsoft Apple à Nearly every modern IDE now supports this Microsoft produced a whole series of products: Visual Basic, Visual C++, … Visual Studio where the “Visual” part was a UI Builder Principle: Visual things should be expressed visually (with minimal code) This turned out to be extremely important in “Making it easy to make things easy” Microsoft Apple Google à Nearly every modern IDE now supports this Microsoft produced a whole series of products: Visual Basic, Visual C++, … Visual Studio where the “Visual” part was a UI Builder Principle: Visual things should be expressed visually (with minimal code) This turned out to be extremely important in “Making it easy to make things easy” Microsoft Apple Google à Nearly every modern IDE now supports this Microsoft produced a whole series of products: Visual Basic, Visual C++, … Visual Studio where the “Visual” part was a UI Builder Principle: Visual things should be expressed visually (with minimal code) This turned out to be extremely important in “Making it easy to make things easy” Microsoft Apple Google à Nearly every modern IDE now supports this Microsoft produced a whole series of products: Visual Basic, Visual C++, … Visual Studio where the “Visual” part was a UI Builder More Successes: Interaction Techniques Dozens (if not 100s) of interaction techniques in the literature “Pinch” gesture (simultaneous scale and translate) More Successes: Interaction Techniques Dozens (if not 100s) of interaction techniques in the literature “Pinch” gesture (simultaneous scale and translate) Dates to Kruger’s 1983 VideoPlace system Speaking of Scale: Zoomable Interfaces (ZUIs) PA3D [Bederson’94] Pioneered using the dimension of scale for interaction à Never run out of space Last Example Touring Machine Project (Columbia) [Feiner et al. ‘97] Last Example Touring Machine Project (Columbia) [Feiner et al. ‘97] Mobile computation Last Example Touring Machine Project (Columbia) [Feiner et al. ‘97] Mobile computation Wireless communications Last Example Touring Machine Project (Columbia) [Feiner et al. ‘97] Mobile computation Wireless communications GPS Last Example Touring Machine Project (Columbia) [Feiner et al. ‘97] Mobile computation Wireless communications GPS Handheld Display & Input (pen rather than touch) Last Example Touring Machine Project (Columbia) [Feiner et al. ‘97] Mobile computation Wireless communications GPS Handheld Display & Input (pen rather than touch) Last Example Touring Machine Project (Columbia) [Feiner et al. ‘97] Mobile computation Wireless communications GPS Handheld Display & Input (pen rather than touch) + See Through Display Last Example Touring Machine Project (Columbia) [Feiner et al. ‘97] Mobile computation Wireless communications GPS Handheld Display & Input (pen rather than touch) + See Through Display Taking a user-centered approach leads to moving from “How do I implement it?”, “What’s the algorithm/architecture?” to “If I had <new