A Multi-Touch Explorer Environment for Eclipse by Daniel Cope A thesis submitted to the Victoria University of Wellington in fulfilment of the requirements for the degree of Masters of Engineering in Software Engineering. Victoria University of Wellington 2013 Abstract The Multi-Touch Explorer Environment (MTEE) is a tool to aid develop- ers during the production of multi-touch enabled applications. The MTEE tool integrates into the Eclipse IDE and can be used to record and play- back user interactions with the program, compare sessions of recorded user interactions and investigate the evolution of the program behaviour. The tool presented in this thesis focuses on the Eclipse IDE and Multi- Touch for Java framework, as they are tools used by both Students and Professional developers. It is demonstrated that the Multi-Touch Explorer Environment can be integrated seamlessly into the Eclipse IDE. It is also demonstrated that the MTEE tool can be used to profile the user’s program with little impact on the performance of both the system or the program itself. ii Acknowledgments Firstly I would like to thank Dr. Stuart Marshall, for supervising the com- pletion of my Masters study. He taught me many times in my undergrad- uate years and supervised the study of my Honours project. I thank him for all of his guidance and support throughout that time. I thank my fellow peers and the members of the HCI group, for pro- viding feedback and support during my project. I wish them the best with all of their studies. I would like to thank my proof readers; Fahmi Abdulhamid, Mata Freshwater, Chris Green and Yohan Ng, for providing much needed feed- back on the clarity of my work. Finally, I thank my parents, David and Anne; my sister, Hannah; my girlfriend, Sam; and all of my friends for encouraging me during my Mas- ters study and all throughout my time at Victoria University. Contents 1 Introduction 1 1.1 Contributions . .2 1.2 Motivation . .3 1.3 Thesis Structure . .3 2 Background & Requirements Analysis 5 2.1 Related Works . .6 2.1.1 User Interaction Recording . .6 2.1.2 User Interface Visualisations . .8 2.1.3 Software Evolution . .9 2.1.4 Visualisations within a Development Environment . 10 2.2 Target Users Profiles . 11 2.2.1 Student Profile . 11 2.2.2 Professional Profile . 16 2.3 Target User Personas . 18 2.3.1 Primary Persona - Student . 18 2.3.2 Secondary Persona - Professional . 21 2.3.3 Negative Persona - Power User . 22 2.3.4 Negative Persona - Beginner Programmer . 23 2.4 Project Requirements . 24 2.4.1 Functional Requirements . 24 2.4.2 Non-functional Requirements . 25 iii iv CONTENTS 3 Design 27 3.1 Tools and Language Choice . 27 3.1.1 Java Language . 28 3.1.2 Tools . 29 3.2 Project Features . 33 3.2.1 Eclipse Plugin . 33 3.2.2 Multi-Touch Explorer Analysis Tool . 41 3.2.3 User Functionality . 41 3.2.4 System Functionality . 46 3.3 Design Trade-Offs . 48 3.3.1 Plugin Environment Setup . 49 3.3.2 MTEA Tool Interface . 49 3.3.3 Component Code View . 51 3.3.4 Component Information View . 52 3.3.5 Performance Overview . 52 3.4 Alternative Designs . 54 3.4.1 Alternative Tools and Languages . 54 3.4.2 Alternative Java Multi-Touch Frameworks . 57 3.5 Design Requirements Analysis . 58 4 Implementation 59 4.1 Language and Tools Critique . 59 4.1.1 Java Language . 59 4.1.2 Eclipse Platform . 60 4.1.3 Multi-Touch 4 Java . 62 4.2 Implementation of Design . 63 4.2.1 Eclipse Plugin . 64 4.2.2 Multi-Touch Explorer Analysis Tool . 69 4.3 Implementation Requirements Analysis . 77 5 Evaluation 79 5.1 Performance Metrics . 80 CONTENTS v 5.1.1 Test Programs . 80 5.1.2 Test System . 83 5.1.3 Testing Method . 84 5.1.4 Limitations . 87 5.1.5 Results . 88 5.1.6 Discussion . 97 5.2 Cognitive Walkthrough . 99 5.2.1 Task Analysis . 100 5.2.2 Walkthrough Results . 104 6 Summary 115 6.1 Performance Metrics . 115 6.2 Cognitive Walkthrough . 118 6.2.1 Proposed Changes . 119 6.3 Future of the Project . 120 6.3.1 Improvements . 121 6.4 Contributions . 121 A XML Data Schema 135 B MTEE Plugin Manifest 139 vi CONTENTS List of Figures 2.1 The Ripples system visualisations . .9 2.2 The Shrimp tool in the Eclipse IDE . 11 3.1 MTE Workspace Layout . 35 3.2 Eclipse Perspective Selection . 38 3.3 Eclipse MTE View Layout . 41 3.4 Component Isolation Mode . 43 3.5 Performance Playback . 46 3.6 Design decisions vs requirements . 58 4.1 Addition of a new perspective . 64 4.2 The MTE Componenent View . 66 4.3 Launching a program from Eclipse . 70 4.4 MTE Tool Software Design Diagram . 71 4.5 Implementation decisions vs requirements . 77 5.1 Slider demo interface . 81 5.2 Multi-Touch demo interface . 82 5.3 3D Multi-Touch gestures demo . 83 5.4 Disk write performance comparison . 92 5.5 Folder size performance comparison . 93 5.6 CPU usage performance comparison . 95 5.7 RAM usage performance comparison . 96 5.8 Average runtime performance comparison . 97 vii viii LIST OF FIGURES 6.1 MTE Workspace Layout w/ changes . 123 Chapter 1 Introduction The Multi-Touch Explorer Environment (MTEE) is a tool to aid develop- ers during the production of multi-touch enabled programs. The tool in- tegrates into the Eclipse IDE to provide developers with a recording and replay feature that allows the developer to record a user’s touch inter- action during testing of their program and play it back. The thesis fo- cuses on the production and assessment of prototype applications using the Multi-Touch for Java framework. This functionality gives the develop- ers the ability to compare the gestures and interactions during prototyping as the UI evolves. A snapshot system allows developers to compare ses- sions from earlier versions of their program, long after the behaviour of the program has changed. MTEE collects data about the developers pro- gram such as; the structure of components in the display, the gestures each component will respond too, and information about gestures that were performed on a component during a recording. This data is displayed directly in the Eclipse IDE. The intention of this project is to show that extra tools can be produced to aid a developer when prototyping multi-touch applications. The MTEE tool presented in this thesis chose to focus on the Eclipse IDE and Multi- Touch for Java, as they are tools used by both students and professionals. The broader initial goal was to develop these tools as a use case for how 1 2 CHAPTER 1. INTRODUCTION multi-touch development can be improved with other frameworks and development environments. This thesis demonstrates that the Multi-Touch Explorer Environment can be integrated seamlessly into the Eclipse IDE. It demonstrates that the MTE tool can be used to profile the user’s program with little impact on the performance of both the system or the program itself. The results of the evaluation imply that the project is a success, and that similar tools could be created to interface with different integrated development envi- ronments and multi-touch frameworks. 1.1 Contributions The completion of the Multi-Touch Explorer Environment (MTEE) resulted in three contributions: • The design of a tool to record and replay user interactions with a multi-touch program. This included the analysis and development of two primary personas related to the target users. • A proof-of-concept prototype for the MTEE. The prototype integrates the MTEE into the Eclipse IDE and focused on projects using the Multi-Touch for Java framework. The MTEE prototype consists of two parts, the plugin for the Eclipse IDE and the Multi-Touch Ex- plorer Analysis (MTEA) tool. • An evaluation of the Multi-Touch Explorer Environment. This in- cluded a set of performance metrics and test suite to allow continual testing of the non-function requirements. It also included a Cogni- tive Walkthrough [22] using expert users, resulting in a list of changes and improvements that can be made to the project going forward. 1.2. MOTIVATION 3 1.2 Motivation Multi-Touch is a relatively new area of mainstream software and hard- ware development. While touch capable devices have existed in research laboratories since the 1980’s [68], the first consumer level products did not start appearing until the late 2000’s with the Apple iPhone [12] and the Mi- crosoft Surface were unveiled [57]. This sudden availability of consumer level hardware quickly made multi-touch an attractive area of software development. For the completion of my Honours project I used the Java program- ming language and the Multi-Touch for Java (MT4J) framework, to pro- duce a system for navigating 3D environments using 2D touch interfaces [23]. I created a series of prototype gestures which could be used to move the user around a 3D virtual scene. Due to the added complexity a multi- touch interface introduced, I found it difficult to compare the behaviour of the different sets of gestures, particularly when looking at ways that the user would actually use them. I also had difficultly distinguishing which components related to which gestures. This led to the idea of creating a dedicated environment for recording and comparing data from multi-touch programs. Such a tool could be ap- plied to any multi-touch development environment, allowing developers to make informed decisions based on data from their application.