Understanding and Supporting End-User Debugging Strategies
Total Page：16
File Type：pdf, Size：1020Kb
AN ABSTRACT OF THE DISSERTATION OF Valentina I. Grigoreanu for the degree of Doctor of Philosophy in Computer Science presented on December 1, 2009. Title: Understanding and Supporting End-User Debugging Strategies Abstract approved: ______________________________________________________ Margaret M. Burnett End users’ programs are fraught with errors, costing companies millions of dollars. One reason may be that researchers and tool designers have not yet focused on end-user debugging strategies. To investigate this possibility, this dissertation presents eight empirical studies and a new strategy-based end-user debugging tool for Excel, called StratCel. These studies revealed insights about males’ and females’ end-user debugging strategies at four levels of abstraction (moves, tactics, stratagems, and strategies), leading to specific implications for the design of tools. Particular results include a set of ten debugging stratagems, which generalized across three environments: the research spreadsheet environment Forms/3, commercial Windows PowerShell, and commercial Microsoft Excel. There was also evidence of the stratagems’ generalization to a fourth set of environments: interactive application design environments, such as Dreamweaver, Flash, and Blend. Males and females statistically preferred and were effective with different stratagems, and females’ stratagems were less well-supported by environments than males’ were. In addition to what stratagems our participants used, we also investigated how these stratagems were used to find bugs, fix bugs, and evaluate fixes. Furthermore, a Sensemaking approach revealed end- user debugging strategies and the advantages and disadvantages of each. We then built StratCel, which demonstrates a strategy-centric approach to tool design by explicitly supporting several of the implications from our studies. StratCel’s evaluation revealed significant benefits of a strategy-centric approach to tool design: participants using the tool found twice as many bugs as participants using standard Excel and fixed four times as many bugs (including two bugs which had not been purposefully inserted by researchers and had gone unnoticed in previous studies). Further, StratCel participants did so much faster than participants using standard Excel. For example, participants using StratCel found the first bug 90% faster and fixed it 80% faster than participants without the tool. Finally, this strategy-based approach helped the participants who needed it the most: boosting novices’ debugging performance near experienced participants’ improved levels. The fact that this approach helped everyone—males and females, novices and experts— demonstrates the significant advantage of strategy-centric approaches over feature-centric approaches to designing tools that aim to support end-user debugging. ©Copyright by Valentina I. Grigoreanu December 1, 2009 All Rights Reserved Understanding and Supporting End-User Debugging Strategies by Valentina I. Grigoreanu A DISSERTATION submitted to Oregon State University in partial fulfillment of the requirements for the degree of Doctor of Philosophy Presented December 1, 2009 Commencement June 2010 Doctor of Philosophy dissertation of Valentina I. Grigoreanu presented on December 1, 2009. APPROVED: Major Professor, representing Computer Science Director of the School of Electrical Engineering and Computer Science Dean of the Graduate School I understand that my dissertation will become part of the permanent collection of Oregon State University libraries. My signature below authorizes release of my dissertation to any reader upon request. Valentina I. Grigoreanu, Author ACKNOWLEDGEMENTS To my advisor, Margaret Burnett, the lessons I have learned from you are indispensible. Thank you especially for throwing me into diverse high-responsibility tasks from the very beginning. There was not a day that passed where I did not learn something new during the whirlwind past years. I will always remember EUD’09: what a great trip, paper, and talks. Margaret Burnett, Paul ElRif, George Robertson, and Jens Mache: Thank you from the bottom of my heart for discovering me and for giving me the opportunity of working with your research groups. Working with you has provided me with a well-rounded view of research that I will continue to cherish as I move forward in my career. I would also like to thank the rest of my committee (Carlos Jensen, Timothy Budd, Maggie Niess, and Bob Higdon) for their thoughtful feedback on my dissertation and, along the way, on my Qualifier and Dissertation Proposal. Thank you for your friendship over the past few years. Thank you also to Paul Cull (the PhD program would not be the same without Friday Coffee Hour) and my other Oregon State professors in the Computer Science, Electrical Engineering, Psychology, and Statistics departments who have all helped create the foundations for this work. Also would like to give many thanks to Ferne Simendinger, Jon Herlocker, Bella Bose, Terri Fiez, and Chris Bell for making me feel welcome at OSU from the very beginning of the program. To all of my coauthors and collaborators: it has been great working with you and I hope we will keep collaborating for years to come. To my new friends (especially OSU EECS graduate students, EUSES consortium members, VLHCC attendants, Microsoft MAX team members, Microsoft Research VIBE and HIP team members, and fellow CHI student volunteers): I have many a fond memory of our chats and your friendship means so much to me. Most importantly, I am eternally grateful to my amazing parents. Nothing would have been possible without your unconditional love during this entire process, and your kind words and deeds. Chester, your energy and enthusiasm have been inspiring. This work was supported in part by Microsoft Research, by NSF grant CNS-0420533, by an IBM Faculty Award, by the EUSES Consortium via NSF grant ITR-0325273, by NSF 0917366, by AFOSR FA9550-09-1-0213, by NSF CCR-0324844, and by the Saturday Academy’s Apprenticeships in Science and Engineering Program for the support of our interns. CONTRIBUTION OF AUTHORS George Robertson helped polish the papers presented in Chapters 5, 7, and 8 and also helped analyze the data for Chapter 5, as the second qualitative analysis researcher, and helped supervise the data collection and analysis for that chapter. Jill Cao, Todd Kulesza, and Chris Bogart helped conduct some of the daily study sessions with participants for the study presented in Chapter 3. Jill and Todd also iterated on and shot the videos used in the new explanations for that study, and analyzed the data related to those new explanations for the Discussion for that chapter. Jill also analyzed the participants’ background data. Chris played a big role during the rebuttal and rewriting phases of the paper submission process. Jill also iterated on an earlier version of Table 19 and created Figure 21. All three helped review the experimental setup and paper write-ups we worked on together. Kyle Rector also helped collect data during daily participant sessions for the study presented in Chapter 3. She also helped as the second content analysis researcher for both that paper and for the study presented in Chapter 6; I used that same data in part in Chapters 7 and 8. Susan Wiedenbeck helped write the related work sections of both Chapters 3 and 6. She also helped review and polish both of the papers in their entirety. Curtis Cook, Sherry Yang, and Vaishnavi Narayanan further reviewed Chapter 3 and gave input along the way about which constructs worked best. Laura Beckwith helped with the write-up for the study presented in Chapter 2, particularly the Introduction and Self-Efficacy Results. The data we mined for the Data Mining Study came from an earlier study of hers. Also, the grade-book spreadsheet task for the studies presented in Chapters 6-8 came from one of her studies, and I harvested the ten bugs I introduced from that particular study’s participants. Xiaoli Fern, with help from Chaitanya Komireddy, led the sequential pattern mining side for the study presented in Chapter 2: chose the algorithm and applied it. They also helped write up the data mining parts of the paper, including the ones about the pattern mining process and the related work. James Brundage was the second content analysis researcher for the study presented in Chapter 4. He also played a significant role in helping set up the study (the task was one of his real-world scripts), run participants, and write a first draft of the section about the script and the bugs. Eric Bahna, Paul ElRif, and Jeffrey Snover also helped review and polish the paper presented in Chapter 4 before submission. Eric wrote about the interaction between stratagems and strategies. Paul helped in reviewing the paper and making sure it adheres to APA standards. He also wrote the first draft of the Participants section in Chapter 4. Jeffrey is PowerShell’s architect and helped write the section of Chapter 4 about the environment. Roland Fernandez and Kori Inkpen also helped with the paper reported in Chapter 5. They supervised the study and helped review and polish off the paper. Discussions with Roland also helped structure Chapter 8. All of the papers which have been accepted to conferences went through a round of peer-review. The reviewers’ comments were often quite insightful and I would therefore like to acknowledge their input here as well, in polishing off the papers. TABLE OF CONTENTS Page 1. Introduction ............................................................................................................................1