Cracking the Cocoa Nut: User Interface Programming at Runtime James Eagan, Michel Beaudouin-Lafon, Wendy E. Mackay To cite this version: James Eagan, Michel Beaudouin-Lafon, Wendy E. Mackay. Cracking the Cocoa Nut: User Inter- face Programming at Runtime. UIST 2011: Proceedings of the 24th ACM Symposium on User Interface Software and Technology, Oct 2011, Santa Barbara, CA, United States. pp.225–234, 10.1145/2047196.2047226. hal-00997886 HAL Id: hal-00997886 https://hal.archives-ouvertes.fr/hal-00997886 Submitted on 2 Jun 2014 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. Cracking the Cocoa Nut: User Interface Programming at Runtime James R. Eagan1,2 Michel Beaudouin-Lafon1,2 Wendy E. Mackay2,1 [email protected] [email protected] [email protected] 1LRI (Univ. Paris-Sud & CNRS); 2INRIA 91405 Orsay Cedex, France ABSTRACT appear spontaneously and in unforeseeable ways [22], even This article introduces runtime toolkit overloading, a novel with the best design practices. As such, there is a gap be- approach to help third-party developers modify the interac- tween the designers’ conception of how software will be used tion and behavior of existing software applications without and its actual use. These gulfs are likely to continue to grow access to their underlying source code. We describe the ab- as software is used in an increasingly rich, ubiquitous, and stractions provided by this approach as well as the mecha- distributed environment. nisms for implementing them in existing environments. We describe Scotty, a prototype implementation for Mac OS X Different approaches have been taken to bridge these gulfs Cocoa that enables developers to modify existing applica- by making software more flexible, including giving users tions at runtime, and we demonstrate a collection of interac- more control over their software’s configuration [16], run- tion and functional transformations on existing off-the-shelf time behaviors [18, 8] and interfaces [21, 9], or by giving applications. We show how Scotty helps a developer make programmers more control over their language [15]. These sense of unfamiliar software, even without access to its source approaches all build on the notion that, although the designer code. We further discuss what features of future environ- may have a good understanding of how to create software ments would facilitate this kind of runtime software devel- that supports the most common use cases, users will push opment. their software in unintended ways. Keywords: user interfaces, meta-toolkits, runtime software Our goal is to bridge the gap between the designed use and development, runtime toolkit overloading actual needs of software by enabling runtime software devel- opment. Runtime software developers are third-party devel- ACM Classification: H.5.m Information interfaces and pre- opers who can modify the behavior and/or interface of soft- sentation (HCI): Misc. ware that nearly satisfies the user’s needs. Runtime devel- opment is concerned with taking the existing behavior of an General terms: Human Factors application and changing it in some way, without requiring INTRODUCTION access to the application source code. Software designers create software with a particular model In this paper, we show how to facilitate this kind of runtime in mind of how it will be used. Even in the most thoughtfully software development in existing applications, we show how designed systems, however, these assumptions may break in it can facilitate new interactions, and we argue for changes to the face of users’ own constraints and demands. Assump- better support it. In particular: tions made by a designer, based on a deep understanding of hardware, software, and human constraints, may change. As • We present runtime toolkit overloading, a novel approach these constraints evolve, so too must the software, often in that provides high-level abstractions to make the redefini- ways not easily reconcilable with its original design. tion of program behaviors a first-class operation, without access to the the application’s source code; Even when these constraints do not evolve, users frequently • We describe how to implement this approach using a set push their software in ways not anticipated by its design- of deep hooks into existing toolkits and runtime systems, ers [20]. A user’s interaction is strongly situated in a par- taking advantage of modern programming languages; ticular context, with particular needs that may arise and dis- • We present Scotty, an implementation of runtime toolkit overloading in the Mac OS X Cocoa environment, and il- lustrate its use to modify a set of off-the-shelf applications; • We provide recommendations to improve support for run- This is the author’s version of the work. It is posted here for your personal time development in future user interface toolkits. use. Not for redistribution. The definitive Version of Record was published in Proceedings of UIST’11, http://dx.doi.org/10.1145/2047196.2047226 RELATED WORK UIST’11, October 16-19, 2011, Santa Barbara, CA, USA. Some applications are designed with modification as an ex- Copyright 2011 ACM. plicit design concern and thus offer some kind of plugin, scripting, or extension mechanism. Most applications, how- WinCuts Façade Prefab SubArctic Scotty ever, are not designed for such extensibility. For these kinds Access to √ √ √ √ √ of applications, other methods are necessary. In this section, Surface Rep we examine different approaches to black-box modification Can change √ √ √ √ of programs (Figure 1). These approaches are black-box in surface rep. the sense that they do not assume that the runtime developer Access to √ √ √ √ has any access to the program source code or to its internal widgets structures. Can change (via I/O (via I/O √ √ widgets redirection) redirection) Scripting, Plugins, and Extensions Access to (only parts that √ Many applications support some degree of extensibility by program model touch toolkit) users through scripting interfaces and/or plugin and exten- Can change (only parts that √ sion mechanisms. Almost all Mac OS applications, for ex- program model touch toolkit) ample, offer a scripting interface (through AppleScript) that Works with √ √ gives control over standard functionality such as launching, multiple platforms quitting, and moving windows [3]. Many web browsers, in- Modifications are typically “safe/ √ √ √ cluding Safari, Firefox, and Chrome, offer plugin and ex- robust” tension interfaces, allowing one to write modifications to the way the browser loads and displays web pages. Scripting and Figure 1: Summary of the capabilities and character- plugin interfaces can provide varying degrees of control, de- istics of different approaches to runtime modification. pending on what internal structures of the application they expose and what, if any, sandboxing they perform to insulate the plugin code from the rest of the application. For exam- isting applications at runtime. They rely on the fact that the ple, the Firefox browser is open enough to enable the Chick- modification environment does not need to know about the enfoot [7] plugin to provide a domain-specific scripting lan- deeper semantics of the application. It suffices to simply guage with deep access to web pages. Similarly, Google’s transport or transform the display and let the user make sense Chrome Frame1 uses Internet Explorer’s plugin system to re- of it. Because no semantic information is available, however, place the rendering engine with that of the Chrome browser. the transformations are therefore limited. These interfaces can potentially offer a high degree of power, Augmented Surface Representations It is possible, though, but only if the application developer has included explicit to augment the surface representation with supplemental se- support for them. Moreover, for complex changes to the mantic information. Fac¸ade [21], for example, integrates program behavior, it is typically necessary to create ad hoc with the X Windows system to gather information about the modifications on a per-application basis. Thus, they may be widget hierarchy. With this additional information, Fac¸ade is of limited utility for defining new behaviors for all applica- able to operate at a finer granularity than WinCuts and can tions. In order to support runtime software development, we even change the behavior of individual widgets. want a general mechanism to write plugins for any applica- tion, regardless of whether it was designed with such support Prefab [9, 10] takes a different approach to augmenting its in mind. understanding of the surface representation of widgets: it uses a vision-based system to recognize widgets using their Changing Program Behavior visual appearance and to create visual stand-ins for them. When the application does not provide explicit support for This has the advantage of being able to operate in a multi- runtime modifications, other approaches are necessary. The platform environment, simultaneously operating on, e. g., na- most basic of these approaches is to operate on the surface tive Windows widgets, Mac OS and X Windows widgets via representation of a window. The surface representation is the a VNC connection, and Flash widgets embedded in a web part of the application directly exposed to the user, such as page. Nonetheless, it can only handle widgets it has been the collection of pixels on the screen, but it could also include trained to recognize. auditory, haptic, or other modalities. All of these approaches face the same limitation: Even with Surface Representations A typical example of this approach supplemental information about, e. g., the widget hierarchy, is VNC [19], which teleports the pixels of an entire desktop widgets are still opaque objects.