http://waikato.researchgateway.ac.nz/ Research Commons at the University of Waikato Copyright Statement: The digital copy of this thesis is protected by the Copyright Act 1994 (New Zealand). The thesis may be consulted by you, provided you comply with the provisions of the Act and the following conditions of use: Any use you make of these documents or images must be for research or private study purposes only, and you may not make them available to any other person. Authors control the copyright of their thesis. You will recognise the author’s right to be identified as the author of the thesis, and due acknowledgement will be made to the author where appropriate. You will obtain the author’s permission before publishing any material from the thesis. Department of Computer Science Hamilton, New Zealand Automating iterative tasks with programming by demonstration Gordon W. Paynter This thesis is submitted in partial fulfilment of the requirements for the degree of Doctor of Philosophy at The University of Waikato. February 26, 2000 © 2000 Gordon W. Paynter Abstract iii Abstract Programming by demonstration is an end-user programming technique that allows people to create programs by showing the computer examples of what they want to do. Users do not need specialised programming skills. Instead, they instruct the computer by demonstrating examples, much as they might show another person how to do the task. Programming by demonstration empowers users to create programs that perform tedious and time-consuming computer chores. However, it is not in widespread use, and is instead confined to research applications that end users never see. This makes it difficult to evaluate programming by demonstration tools and techniques. This thesis claims that domain-independent programming by demonstration can be made available in existing applications and used to automate iterative tasks by end users. It is supported by Familiar, a domain-independent, AppleScript-based programming-by-demonstration tool embodying standard machine learning algorithms. Familiar is designed for end users, so works in the existing applications that they regularly use. The assertion that programming by demonstration can be made available in existing applications is validated by identifying the relevant platform requirements and a range of platforms that meet them. A detailed scrutiny of AppleScript highlights problems with the architecture and with many implementations, and yields a set of guidelines for designing applications that support programming-by-demonstration systems and other agents. An evaluation shows that end users are capable of using programming by demonstration to automate iterative tasks. However, the subjects tended to prefer other tools, choosing Familiar only when the alternatives were unsuitable or unavailable. Familiar’s inferencing is evaluated on an extensive set of examples, highlighting the tasks it can perform and the functionality it requires. Acknowledgments v Acknowledgments I sometimes wonder what I would be doing now if Ian Witten had not talked me into writing a thesis. (I imagine I would have cashed in my stock options and retired by now.) I also wonder what might have become of me had I attempted a thesis under a supervisor less generous with his time and experience than Ian, without whose encouragement and example I surely would never have finished. Thanks Ian. My two other supervisors, Geoff Holmes and Mark Apperley, have been very supportive, and patient. I’d like to thank them, and those others who have read this thesis and offered advice: Sally Jo Cunningham, Steve Jones, Simon Christiansen, Stuart Yeates, and Grant Floyd; and also my mother (who made one helpful suggestion before falling asleep at Section 1.3). I’m grateful to the many people who have commented on, and contributed to, my research over the last four years, including Allen Cypher, Eibe Frank, Craig Nevill-Manning, Carl Gutwin, Henry Lieberman, the staff of my department, and numerous others. I have borrowed code and ideas from any number of people, including Tom Bonura, Roman Zeiliger, Marty Geier (and his colleagues at Mitre), and the Weka programmers. I am sure I have omitted numerous others; for this I apologise. I would surely not have lasted four years without the diversions afforded me by my flatmates: Simon, Sarah, Dale, Teri, Donna, Cat, Nick, Jan, Elliot, and others too numerous to mention. Our department is blessed with a fantastic soccer team, comprising many of our students and staff. Thanks Alvin (who started it all), Annette, Aziz, Carl, Chithiran, Conrad, David B., David M., Dong, Eibe, Geoff, Jamie, Len, Max, Kim, Richard, Rob, Steve, Stuart, Talia, Tony, Yong… Finally, I’d like to thank my parents, my brother Craig, my sister Sharon, and the rest of my family, particularly the Gordons, for their support and encouragement. I dedicate this thesis, and the years of work it represents, to my uncle Doug Gordon, without whose support it could never have been attempted, let alone finished. Table of contents vii Table of contents Abstract iii Acknowledgements v List of Figures xv List of Tables xix Chapter 1 Introduction 1 1.1 Programming by demonstration (PBD) ...............................................1 1.2 Thesis statement.......................................................................................3 1.2.1 Automating iterative tasks...................................................4 1.2.2 End users and applications ..................................................5 1.2.3 Domain independence..........................................................5 1.3 Examples of iterative tasks.....................................................................6 1.3.1 Image transfer and conversion ............................................7 1.3.2 Averaging column data ........................................................8 1.3.3 Indexing document files .......................................................9 1.4 Thesis structure......................................................................................10 viii Table of contents Chapter 2 Background 13 2.1 Definitions and usage........................................................................... 14 2.2 Automating iteration............................................................................ 21 2.3 PBD and iteration problems ................................................................ 24 2.4 PBD and repetition problems.............................................................. 27 2.4.1 Macro recorders .................................................................. 28 2.4.2 Extending macro recorders ............................................... 28 2.4.3 Editable histories................................................................. 30 2.5 Other demonstrational interfaces....................................................... 31 2.6 Discussion and criticism of PBD......................................................... 34 2.6.1 Problems with agents ......................................................... 34 2.6.2 Problems with PBD............................................................. 35 2.6.3 Challenges for PBD research............................................. 37 2.7 Summary ................................................................................................ 38 Chapter 3 Design considerations 39 3.1 Existing PBD designs............................................................................ 40 3.1.1 Feature-oriented designs ................................................... 41 3.1.2 User-oriented designs......................................................... 41 3.1.3 Successful end-user programming................................... 43 3.2 PBD for the end user............................................................................. 44 3.2.1 End user motivations.......................................................... 44 3.2.2 End user abilities................................................................. 45 3.2.3 End user attitudes ............................................................... 45 3.2.4 Design guidelines................................................................ 47 3.3 The Familiar user interface.................................................................. 51 3.3.1 Arranging files..................................................................... 52 3.3.2 Arranging files when Familiar makes an error .............. 54 3.3.3 Sorting files .......................................................................... 56 3.3.4 Converting images.............................................................. 59 3.3.5 Converting images with the evaluation interface .......... 61 3.4 Familiar’s limitations............................................................................ 63 3.5 Summary ................................................................................................ 64 Table of contents ix Chapter 4 The Familiar architecture 65 4.1 System overview....................................................................................67 4.2 The event recorder.................................................................................70 4.2.1 AppleScript...........................................................................70 4.2.2 Application knowledge ......................................................71 4.2.3 Background knowledge......................................................76 Application independence ..............................................................77