WADE: Simplified GUI Add-on Development for Third-party Software Xiaojun Meng, Shengdong Zhao, Yongfeng Huang, Zhongyuan Zhang, James Eagan, Ramanathan Subramanian To cite this version: Xiaojun Meng, Shengdong Zhao, Yongfeng Huang, Zhongyuan Zhang, James Eagan, et al.. WADE: Simplified GUI Add-on Development for Third-party Software. CHI ’14: ACM SIGCHI Conference on Human Factors in Computing Systems, Apr 2014, Toronto, Canada. pp.10, 10.1145/2466110.2466192. hal-01115276 HAL Id: hal-01115276 https://hal-imt.archives-ouvertes.fr/hal-01115276 Submitted on 13 Sep 2019 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. WADE: Simplified GUI Add-on Development for Third-party Software Xiaojun Meng1, Shengdong Zhao1, Yongfeng Huang1, Zhongyuan Zhang1, James R. Eagan2, Ramanathan Subramanian3 1NUS-HCI Lab, National University of Singapore, 2Télécom ParisTech & CNRS LTCI UMR 5141, 3ADSC, University of Illinois at Urbana-Champaign 1{xiaojun,zhaosd}@comp.nus.edu.sg,[email protected],[email protected] [email protected],[email protected] Figure 1: WADE overview: 1) Kevin wants to make the following modifications to the original Paint.NET interface i) change menu labels from English to Chinese, ii) remove unused menus and icons, and iii) add a new “undo all” function. To this end, he 2) Installs WADE in Paint.NET, 3) Clones the Paint.NET GUI into the GUI builder of a WADE-supported IDE. Then, he (a) modifies the GUI using a WYSIWYG editor, and (b) writes code associated with the “undo all” widget via the event handler template. 4) All the above changes are compiled to an add-on that can be installed into Paint.NET for easy and convenient future use. ABSTRACT ACM Classification Keywords We present the WADE Integrated Development H.5.2. Information Interfaces and Presentation: User Environment (IDE), which simplifies the modification of Interfaces the interface and functionality of existing third-party General Terms software without access to source code. WADE clones the Human Factors Graphical User Interface (GUI) of a host program through dynamic-link library (DLL) injection in order to enable (1) INTRODUCTION WYSIWYG modification of the GUI and (2) the Software rarely fulfills the needs of all users all the time [7, modification of software functionality. We compare WADE 12]. Mindful of the need to make software adaptable to with an alternative state-of-the-art runtime toolkit individual needs, developers typically allow for software overloading approach in a user-study, finding that WADE customization by providing: significantly simplifies the task of GUI-based add-on • Capabilities for reconfiguring existing features and development. functions to suit personal taste (e.g., via preferences Author Keywords panes or dot files), or WADE; GUI; Add-on Integration; WYSIWYG; IDE • Software architecture for incorporating add-ons (e.g., using plugins, scripts and/or extensions) to enhance/modify the behavior of the original application. While these approaches can provide users with a great deal of control, every approach necessitates additional effort CHI 2014, April 26–May 1, 2014, Toronto, ON, Canada. from the software developers to explicitly provide Copyright © 2014 ACM. This is the author's version of the work. It is customization support at the software development stage. posted here for your personal use. Not for redistribution. The definitive Version of Record was published in Proceedings of the ACM For example, plugins, scripting interfaces and extensions SIGCHI Conference on Human Factors in Computing Systems. require the developer to provide and maintain an external http://dx.doi.org/10.1145/2556288.2557349 API to their software, which may potentially require maintaining an additional and separate interface to internal associate event handlers to existing widgets, so that functionality. enhancing/modifying software behavior becomes simpler. Owing to the above issues, many software developers do Figure 1 shows an example add-on development scenario not provide support for add-ons. Even when they do, such using WADE. Currently, WADE supports add-on support is often limited [1]. To address this limitation, development using both the open source SharpDevelop 4.2 much research has focused on approaches that enable third- and the Microsoft Visual Studio 2012 Ultimate IDEs for party developers to modify the interface or behavior of Windows Form applications on the Windows XP and existing applications without access to source code or to an Windows 7 platforms. external API. These approaches typically work by either: 1) We conducted an experiment to compare WADE with a operating on the surface-level of the interface, intercepting Scotty-like toolkit-based deep approach for modifying third input events and output pixels before they are delivered to party applications. Our results show that users subjectively the application (e.g., Prefab [2, 3], Façade [14]), or 2) found WADE much easier to use, and were objectively able integrating with the toolkit to gain access to the internal to develop GUI-based modifications 2.4 times faster than program structures (e.g., Scotty [4], SubArctic [5]). For the alternative approach on average. To summarize, the convenience, we call the former as surface-based contributions of this work are: approaches and the latter as toolkit-based deep approaches. Surface-based approaches allow modifications to GUI • We present the WADE prototype along with its software elements without access to the internal structure of an architecture as an integrated solution for significantly application. For example, Façade allows for reconfiguring facilitating add-on creation for third party software GUI elements via a simple drag and drop interface [14]. without source code. However, such approaches are limited by their ability to • The WADE IDE provides scaffolding for code-based infer the structure and functionality of the interface because GUI modification through template generation, thereby they do not have access to the internal program objects or enabling robust implementation of the complex their semantics. E.g., adding new functionality or boilerplate associated with runtime modification. modifying the behavior of a GUI widget is difficult to • We present the results of an empirical comparison accomplish using surface-based approaches [4]. between WADE and the state-of-the-art Scotty approach This limitation can be overcome to some extent by toolkit- for modifying software [4], which shows that WADE is based deep approaches such as Scotty [4] or SubArctic [5], significantly faster for GUI modifications. which operate below the surface of the program to reveal RELATED WORK the underlying program logic and functionalities. This As previously mentioned, surface-based adaptation [2, 3, allows them to alter the system’s appearance and behaviors 14, 15] and toolkit-based subsurface modification [4, 5, 16] beyond the surface level. However, toolkit-based deep are the two main approaches that support third-party approaches can be challenging to use. They require a application modifications without access to the software’s thorough understanding of the relevant parts of the system source code. As a comprehensive review of the different in order to realize the desired behavior. Even for variants of these two approaches has already been discussed experienced developers, much effort is needed to make in Eagan et al. [4], we now highlight those works most relatively simple modifications to third party software. relevant to WADE. Therefore, there exists a trade-off between generalizability, Surface-level modification ease of use, and power (the ability to perform deeper Surface-level modifications do not require any support by modifications). While all previous approaches have their the application developer. Instead, they operate on the advantages, additional solutions are still needed to better interface that is presented to the user and the input events balance the power and ease of use for runtime modification he or she provides. For example, Yeh et al.'s Sikuli of third-party software. scripting environment [1] allows users to write scripts that In this paper, we propose WADE, a simplified and reference screenshots of particular controls, to refer to WSYWYG Add-on Development Environment that can existing application elements. ease the task of modifying GUI-based functions in existing Stuerzlinger et al.'s UI Façades [14] intercept individual software with or without source code, while still enabling widgets as they interact with the window server. This developers to make deep changes to the software behavior. allows a developer to easily replace them at the window To achieve this, WADE injects a dynamically-linked library server level with an alternate implementation, such as by (DLL) into the host program to retrieve the GUI hierarchy regrouping together widgets from different applications or of the host program. It then clones the interface in the IDE replacing a radio button with a pop-down menu. Dixon and so that properties of GUI elements can be directly modified. Fogarty's Prefab [2] examines pixels as they are drawn on Furthermore, WADE provides scaffolding to directly the screen to infer which parts correspond to which widgets. It then allows the interception and replacement of these pixels to change the output of a particular interface. 2. He then exports those changes to a new add-on Combined with input redirection, Prefab can enable component that Ivanov can load into his English copy. alternate software functionality. However, all of these solutions are limited by their ability to infer the structure and functionality of the interface.