The University of Nottingham School of Computer Science A Framework for Interactive End-user Web Automation Essam Eliwa Thesis submitted to the University of Nottingham for the degree of Doctor of Philosophy February 2013 To my parents ii Abstract This research investigates the feasibility and usefulness of a Web-based model for end-user Web automation. The aim is to empower end users to automate their Web interactions. Web automation is defined here as the study of theoretical and practical techniques for applying an end-user programming model to enable the automation of Web tasks, activities, or interactions. To date, few tools address the issue of Web automation; moreover, their functionality and usage are limited. A novel model is presented, which combines end-user programming techniques and the software tools philosophy with the vision of the “Web as a platform.” The model provided a Web-based environment that enables the rapid creation of efficient and useful Web-oriented automation tools. It consists of a command line for the Web, a shell scripting language, and a repository of Web commands. A framework called Web2Sh (Web 2.0 Shell) has been implemented, which includes the design and implementation of scripting language (WSh) enabling end users to create and customise Web commands. A number of Web2Sh-core Web commands were implemented. There are two techniques for extending the system: developers can implement new core Web commands, and the use of WSh by end users to connect, customise, and parameterise Web commands to create new commands. The feasibility and the usefulness of the proposed model have been demonstrated by implementing several automation scripts using Web2Sh, and by a case study based experiment that was carried out by volunteered participants. The implemented Web2Sh framework provided a novel and realistic environment for creating, customising, and running Web-oriented automation tools. iii All praise be to Allah, the most gracious, the most merciful Acknowledgements It is hard to acknowledge everybody that helped me to start, continue, and finish this work. I attempt my best and apologise dearly to those I did not mention. I would like to extend particular thanks to my supervisors, Dr. Colin Higgins and Dr. Peter Blanchfield, for their support, guidance, and comments on this thesis, and for their moral support over the time of my study. Foremost, I express my deep gratitude to my dear friends Ashraf AbdelRaouf, Mohamed Sabri, Rami Ghorab, P. Christofi, A. Soaifan, I. Balčaitė, and M. Al- Momani for all their support in many aspects during my study years. The help of a number of computer science researchers has been essential at the early years of my studies, so special thanks to A.Tsintsifas, S.Bagley, I.Celik, M.Meccawy, M.Begum, S. Myo Htwe, and C.Schoreels. The Phd for an international student is a lonely journey without great friends. I was lucky to meet so many of them in Nottingham; they have been like a second family away from home, so special thanks to A.Sabra, A.Khalifa, M.Mansour, A.Freewan, A.Basiouni, N.Wahba, Y.Saber, M.Sobhi, S.Abdulkadir, M.Ali, A.Soghier, K.Saleh, G.Frisso, L.McCormack, J.Durkin, A.Galati, E.Bachis, and A.Mangood. Special gratitude to all the Egyptians who have participated in the 25th January revolution, mostly the ones who lost their lives in all of Egypt’s Tahrir squares. There are no adequate words to express my gratitude to my parents for supporting me emotionally, spiritually, and financially. I am also forever thankful to Ayman and Ola Eliwa for being great and supportive siblings in my most difficult moments. I am indebted to my whole family, especially to Dr. Hesham Khalil, my cousin who encouraged and inspired me since I was a very young. Finally, sincere love and gratitude to my wife Halah Yasser, whom I was blessed to marry, and who supported me to complete such a long life-changing journey. Thank you all, Essam Eliwa iv Table of Contents Abstract ............................................................................................................................... ii Acknowledgements ............................................................................................................ iii Table of Contents ................................................................................................................ iv List of Tables ................................................................................................................... viii List of Figures and Illustrations .......................................................................................... ix List of Symbols, Abbreviations, and Nomenclature ........................................................... xi CHAPTER ONE: INTRODUCTION .................................................................................. 1 1.1 Motivation ................................................................................................................. 3 1.2 Background ................................................................................................................ 5 1.3 End-user Web Automation ...................................................................................... 12 1.4 Research Issues ........................................................................................................ 14 1.4.1 Scope ............................................................................................................... 14 1.4.2 General Aims and Objectives .......................................................................... 15 1.4.3 Specific Objectives .......................................................................................... 17 1.4.4 Potential Users ................................................................................................. 17 1.4.5 Approach ......................................................................................................... 19 1.5 Contributions ........................................................................................................... 20 1.5.1 Web2Sh Framework Overview ....................................................................... 20 1.6 Organisation of the Thesis ....................................................................................... 22 CHAPTER TWO: RELATED WORK ............................................................................. 24 2.1 Introduction ............................................................................................................. 24 2.2 World Wide Web ..................................................................................................... 24 2.2.1 A Brief History of the Web ............................................................................. 26 2.2.2 Web Architecture ............................................................................................ 28 2.2.2.1 Identification .......................................................................................... 29 2.2.2.2 Interaction .............................................................................................. 29 2.2.2.3 Representation ....................................................................................... 29 2.2.3 Web Scripting Languages ............................................................................... 31 2.2.4 Web Mining ..................................................................................................... 33 2.2.5 Semantic Web ................................................................................................. 33 2.2.6 Syndication Feeds ........................................................................................... 35 2.2.7 Web 2.0 ........................................................................................................... 36 2.2.8 Rich Internet Applications (RIA) .................................................................... 38 2.2.9 User-Generated Content (UGC) ...................................................................... 39 2.3 Web Automation Tools ........................................................................................... 40 2.3.1 Web SPHINX .................................................................................................. 45 2.3.2 YubNub ........................................................................................................... 46 2.3.3 Mashups .......................................................................................................... 47 2.3.4 Yahoo Pipes ..................................................................................................... 49 2.3.5 Microsoft Popfly ............................................................................................. 51 2.3.6 Firefox plugins ................................................................................................ 52 2.4 UNIX Shell Scripting .............................................................................................. 54 2.5 A Programmable Web ............................................................................................. 56 2.6 Summary .................................................................................................................. 57 v CHAPTER THREE: WEB2SH FRAMEWORK DESIGN .............................................. 58 3.1 Introduction ............................................................................................................. 58 3.2 Design Objectives and Requirements .....................................................................