Blaze – Automating User Interaction in Graphical User Interfaces
Total Page：16
File Type：pdf, Size：1020Kb
Blaze – Automating User Interaction in Graphical User Interfaces Gabriel José Faustino Fonseca Barata Dissertation for the Master’s degree in Information Systems and Computer Engineering Jury Chairman: Doctor Joaquim Armando Pires Jorge Adviser: Doctor Daniel Jorge Viegas Gonçalves Members: Doctor Carlos Alberto Pacheco dos Anjos Duarte November 2009 ACKNOWLEDGEMENTS From the bottom of my heart, I would like to thank my parents, Ramiro and Maria do Rosário, for being so supportive and always being there for me. Thank you so very much for always encouraging me to go one step further. A special thanks to my uncle José and my cousin Sofia, for being supportive and for always worrying about me and my future. My most heartfelt thanks to Alexandre Ramos, Filipe Larsson, Paulo Carrilho and Sérgio Mendeiros, for being like brothers to me and even sacrifice yourselves for me and my work. Thanks a lot to Daniela Loio for being there for me and supporting me in my long and tortuous testing sessions. Thanks to Ana Rodrigues, Magda Carvalho and Sandra Gama, for being good friends and believing in my effort. Thanks Tiago Guerreiro for being such a good co-adviser and supporting me in this research. I am especially grateful to my adviser, Daniel Gonçalves, for advising and guiding me through this tortuous path that now comes to an end. To all of these people aforementioned I am grateful for always believing in me. Thanks to all users that accepted to collaborate with me and perform user tests. Each testing session lasted for two hours and I know it was quite a sacrifice. Thanks a lot for your time. At last, but no least, I would like to thank the whole community for receiving Blaze so well and for the wonderful reviews. I hope we can work together in order to keep Blaze getting better and better. I RESUMO Os utilizadores dos sistemas operativos actuais são constantemente confrontados com tarefas repetitivas, que surgem do uso diário. Apesar de actualmente existirem vários application launchers que tentam automatizar várias tarefas bem conhecidas, estes não estão à altura da unicidade das tarefas repetitivas que surgem do uso corrente. Muitas vezes, para tentarem maximizar o seu desempenho, os utilizadores tentam soluções alternativas, como o uso de linguagens de scripting ou mesmo editores de macros. Contudo, este tipo de solução está fora do alcance do utilizador comum, obrigando-os a completar a tarefa à mão. Apresentamos um sistema inovador, denominado Blaze, que é capaz de automatizar as tarefas repetitivas do utilizador, não só numa única aplicação, mas ao nível do sistema operativo. A nossa abordagem baseia-se na monitorização contínua e passiva das actividades do utilizador e no estabelecimento de relações entre as acções capturadas. Um algoritmo de procura por prefixos comuns em árvores de sufixos é utilizado para detectar padrões repetitivos no historial de acções do utilizador, sem interromper o utilizador. Consequentemente, os padrões repetitivos detectados, em conjunto com as relações estabelecidas, permitem inferir o comportamento futuro do utilizador e compor acções que completem o resto da tarefa repetitiva. Estas acções podem ser guardadas em scripts que têm a capacidade de se adaptarem a novas situações, diferentes daquelas em que as acções foram inicialmente detectadas. Para além disso, o Blaze leva o conceito de application launcher para um novo nível, introduzindo um algoritmo de predição textual que consegue tolerar erros de escrita. Os testes efectuados mostram que o Blaze permite ao utilizador efectuar tarefas repetitivas 3 vezes mais rápido sem sacrificar significativamente o desempenho do computador. PALAVRAS-CHAVE Interfaces Adaptativas, Programação Implícita por Exemplo, Detecção de Padrões Repetitivos, Automatização de Tarefas, Sensibilidade ao Contexto do Utilizador. II ABSTRACT Every day, operating system users are faced with repetitive tasks that should be automated. Although there are several application launchers currently available that can automate well known repetitive tasks, most of those tasks are unique and arise from everyday usage, making them unpredictable. In a situation where users face a repetitive task, they have only a few choices: if they are experienced users, they’ll probably be able to build a script to automate the task; if not, they will need to complete the remainder of the task by themselves or try to find a special-purpose application to do it and spend as much time learning how to use it as they would spend finishing it by hand. We present Blaze, a system capable of automating user’s repetitive tasks in graphical user interfaces, operating system-wide. Blaze consists of an application launcher, which enhances its counterparts’ features, and is continuously monitoring and analyzing the user’s activities, without disturbing him. Thereby, Blaze identifies relationships and repetitions in the users’ actions and offers to accomplish the remaining of the task, in their place. To detect repetitions, Blaze resorts to a fast string search algorithm, based on common prefix search in a suffix-tree. Furthermore, Blaze is able to abstract a suggested automation and export it to a general-purpose script, which can later be replayed, even in a different situation from the one in which the repetition was detected. Our tests show that users can perform recurrent tasks 3 times faster using Blaze, without significantly sacrificing the computer’s performance. KEYWORDS Programming by Example, Adaptive Interfaces, Repetitive Pattern Detection, End-User Programming, User Context Awareness, Task Automation. III TABLE OF CONTENTS 1 Introduction ................................................................................................................................................................ 1 1.1 Contributions .................................................................................................................................................... 3 1.2 Publications ....................................................................................................................................................... 3 1.2.1 Conferences ............................................................................................................................................. 3 1.2.2 Dissemination ......................................................................................................................................... 4 1.3 Document Structure ...................................................................................................................................... 4 2 Related Work.............................................................................................................................................................. 5 2.1 Application Launcher Analysis.................................................................................................................. 5 2.1.1 Ac’tivAid .................................................................................................................................................... 5 2.1.2 CAL .............................................................................................................................................................. 6 2.1.3 Colibri ......................................................................................................................................................... 6 2.1.4 Dash Command ...................................................................................................................................... 6 2.1.5 Enso Launcher ........................................................................................................................................ 7 2.1.6 Executor .................................................................................................................................................... 7 2.1.7 Find and Run Robot.............................................................................................................................. 8 2.1.8 Google Desktop ...................................................................................................................................... 8 2.1.9 Keybreeze ................................................................................................................................................. 9 2.1.10 Launchy ..................................................................................................................................................... 9 2.1.11 Shuriken ................................................................................................................................................. 10 2.1.12 Skylight ................................................................................................................................................... 10 2.1.13 Speed Launch ....................................................................................................................................... 11 2.1.14 WinSilver ............................................................................................................................................... 11 2.1.15 Dmenu ..................................................................................................................................................... 12 2.1.16 Gmrun ..................................................................................................................................................... 12 2.1.17