2010 IEEE Symposium on Visual Languages and Human-Centric Computing A Debugging Perspective on End-User Mashup Programming Jill Cao1, Kyle Rector1, Thomas H. Park2, Scott D. Fleming1, Margaret Burnett1, Susan Wiedenbeck2 1Oregon State University 2Drexel University {caoch, rectorky, sdf, burnett}@eecs.oregonstate.edu {tp352, sw53}@drexel.edu Abstract Our first goal was to understand the debugging In recent years, systems have emerged that enable strategies that end users use in programming mashups. end users to “mash” together existing web services to Prior studies on debugging have identified debugging build new web sites. However, little is known about strategies used by end users in spreadsheet [16] and how well end users succeed at building such mashups, scripting environments [7]. Understanding such strate- or what they do if they do not succeed at their first at- gies has provided valuable insights into how such envi- tempt. To help fill this gap, we took a fresh look, from ronments can be improved to support debugging strat- a debugging perspective, at the approaches of end us- egies [16]. ers as they attempted to create mashups. Our results Our second goal was to understand the impact of reveal the end users’ debugging strategies and strategy gender differences on how end users debug mashups. barriers, the gender differences between the debugging Prior work in the context of spreadsheet and scripting strategies males and females followed and the features environments has shown that gender differences can they used, and finally how their debugging successes play an important role in the adoption of strategies [7, and difficulties interacted with their design behaviors. 16] and the usage of features that support the strate- gies [2]. Understanding such differences has revealed 1. Introduction gender biases and informed tool designs that narrowed the gender gap while benefitting both males and fe- Mashup programming is a relatively new paradigm males [6]. We seek to extend these findings to mashup in end-user programming. Mashups repurpose and environments. combine existing web content and services to meet the Our third goal was to understand how debugging specific needs of the user. For instance, an avid news interacts with design when end users create mashups. follower might build a mashup that combines news Our prior study [3] revealed that instances of design feeds from MSN, New York Times, and Fox, and that permeated the end users’ programming processes while filters the content based on his favorite topics (e.g., creating mashups. Furthermore, viewing mashup pro- world news and business). There are a wide variety of gramming through the design lens produced promising mashup programming environments, such as Microsoft design implications for mashup environments. In this Popfly, Intel Mash Maker, and Yahoo! Pipes. Howev- work we seek to extend these findings and implications er, little is known about how well these environments by focusing on debugging. support the “mashing” needs of end users. Our work Given these three goals, our investigation sought to seeks to fill this knowledge gap. answer the following research questions: While conducting a recent study of end users build- RQ1: What debugging strategies do end users adopt ing mashups [3], we noticed that participants spent a when they are creating mashups? What bar- substantial amount of time (76.3%) debugging. This riers do they encounter? observation suggests that they tended to “debug into existence”—that is, they created their mashups through RQ2: Are there gender differences in end users’ a process of successive refinement, which was driven usage of debugging strategies in mashups? by the finding and fixing of bugs [13]. Given the pre- RQ3: How does debugging interact with design when valence of debugging activity, we believe that effective end users are creating mashups? debugging support is critical for successful mashup environments. 2. Related Work Therefore, in this paper, we report on a new analy- Despite the proliferation of mashup environments, sis of that study, from the perspective of end users’ research has found that end users’ lack of technical debugging behaviors and barriers while creating ma- expertise is a serious barrier to creating mashups [10, shups. Our investigation had three main goals. 19]. In response to this problem, researchers have pro- 978-0-7695-4206-5/10 $26.00 © 2010 IEEE 149 DOI 10.1109/VLHCC.2010.29 posed mashup environments (e.g., Vegemite and Ma- Procedure. The study used the think-aloud method, so shroom) that reduce such barriers by using techniques we conducted it one participant at a time. Participants such as direct manipulation, programming-by- first filled out a background questionnaire and worked demonstration, and intuitive nested-table data struc- on a 20-minute tutorial that familiarized them with tures. Usability studies have demonstrated that such Popfly. They then completed a self-efficacy question- environments can significantly benefit end users. naire [4], which we adapted to the task of mashup crea- A few studies have explored end-user perceptions tion. Participants then practiced “thinking aloud” be- of mashups systems. In a survey of end users [22], fe- fore proceeding to the main task. males expected mashup creation to be more difficult The task involved creating a mashup to include (1) than did males, suggesting gender differences in self- a list of local theaters, (2) the movies being shown at efficacy. To inform the design of mashup tools, Zang each theater along with information such as show and Rosson [23] investigated end users’ mental models times, (3) a picture for each movie, and (4) a news sto- of mashups—for example, how users thought about ry about each movie. Participants had 50 minutes to relationships in online data, and how they might like to complete the task. If they were unable to make integrate data. Jones and Churchill [8] studied the on- progress for 15 minutes, the researcher administered an line forums of Yahoo! Pipes users to understand how additional 5-minute mini-tutorial. Although this tutori- end users collaborate to resolve problems. al may have influenced participants’ behaviors, we Our investigation complements the above research relate its effect to encountering a well-chosen example. by investigating how users go about debugging while Half of the participants (2 males, 3 females) received building mashups. The investigation differs from the the mini-tutorial, and it proved helpful to three of them existing literature on end-user debugging (e.g., [7, 16]) (2 males, 1 female). We collected video data (including in two ways. First, prior studies asked participants to facial expressions) as well as the final mashups. find and fix seeded defects in spreadsheets or scripts. Our study asked participants to create mashups; thus, 4. Analysis Methodology debugging was not the central goal. Second, our inves- Participants debugged intermittently as they created tigation is the first to look at strategies, design beha- their mashups. Thus, we first identified sections of the vior, and gender differences in the debugging of ma- transcripts in which participants were debugging (i.e., shups. attempting to correct problems they had introduced). We then performed a protocol analysis on those sec- 3. Empirical Study tions. This investigation entailed a new analysis of data Grigoreanu et al.’s collection of debugging strate- from a previous study [3] in which we observed ten gies [7] served as the basis for our strategy analysis participants engaged in a mashup-creation task using (Table 1). Two researchers independently coded all Microsoft Popfly Mashup Creator. transcripts with the strategy codes, then jointly re- Participants. Participants comprised four female and six male college students from a wide variety of majors (e.g., biology, nutrition science). None were CS stu- dents, and none had taken CS courses beyond the ele- mentary level. One female and four males had past programming experience either in high school, college, or both (no more than two courses). Popfly Mashup Creator. In Popfly, users build ma- shups in a dataflow style using blocks. A block per- forms a set of parameterized operations such as data retrieval and data display. Users connect blocks to route the output of one block into the input of another. Figure 1 shows a Flickr block sending images and geographical coordinates of “beaches” to a Virtual Earth block (top, middle) to display the images on a map (bottom). Popfly users can make their projects public and access others’ public projects. (More details of Popfly are given later.) Figure 1. Example mashup in Popfly. Top: the blocks. Mid- dle: block settings. Bottom: the running mashup. 150 solved any inconsistencies. spection co-occurred the most (99 participant minutes), Framing captures a designer’s efforts to understand agreeing with previous results [16]. Other frequent co- and define a design problem, and determines the boun- occurrences were Dataflow and Testing (58), and Code daries for designers’ actions in solving the problem Inspection and Dataflow (53). [14]. Designers frame a problem by identifying areas In the remainder of this section, we focus on Code of the solution space that they want to explore [5]. Inspection, Dataflow, and Help, which were the strate- When designers choose an unworkable frame, they gies in which barriers arose. We also describe an anti- must reframe, which involves assessing and then mod- barrier associated with Feedback Following. ifying or replacing the current frame. Code Inspection: Getting eyes on the code. In Popf- As a conservative approach to coding reframing, we ly, Code Inspection involved scrutinizing the mashup coded only those instances that were evidenced by ac- logic, such as block settings and inter-block connec- tions. Two types of actions indicated reframing: (1) tions. Participants encountered two prominent barriers adding/removing a block, and (2) adding/removing in applying this strategy. First, Popfly blocks are connections between blocks.