DOCSLIB.ORG

How to Recruit Participants for Usability Studies

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
How to Recruit Participants for Usability Studies 234 Tips and Tricks for Recruiting Users as Participants in Usability Studies By Deborah Hinderer Sova and Jakob Nielsen 48105 WARM SPRINGS BLVD. FREMONT, CA 94539–7498 USA WWW.NNGROUP.COM Copyright © Nielsen Norman Group. All Rights Reserved. Get your own copy from: www.nngroup.com/reports/how-to-recruit-participants-usability-studies About This Free Report This report is a gift for our loyal audience of UX enthusiasts. Thank you for your support over the years. We hope this information will aid your efforts to improve user experiences for everyone. The research for this report was done in 2003, but the majority of the advice may still be applicable today, because people and principles of good design change much more slowly than computer technology does. We sometimes make older report editions available to our audience at no cost, because they still provide interesting insights. Even though these reports discuss older designs, it’s still worth remembering the lessons from mistakes made in the past. If you don’t remember history, you’ll be doomed to repeat it. We regularly publish new research reports that span a variety of web and UX related topics. These reports include thousands of actionable, illustrated user experience guidelines for creating and improving your web, mobile, and intranet sites. We sell our new reports to fund independent, unbiased usability research; we do not have investors, government funding or research grants that pay for this work. Visit our reports page at https://www.nngroup.com/reports/ to see a complete list of these reports. HOW TO SHARE Do not link directly to the PDF file (the hosted address could change). Instead, we encourage you to distribute the following link to the report’s page on our website to allow people to decide whether to download it themselves: www.nngroup.com/reports/how-to-recruit-participants-usability-studies © NIELSEN NORMAN GROUP WWW.NNGROUP.COM 2 Copyright Notice Please do not post this document to the internet or to publicly available file-sharing services. This report is free, but it is still copyrighted information that may be updated from time to time, so please don’t distribute this file or host it elsewhere. Even when people post documents with a private URL to share only with a few colleagues or clients, search engines often index the copy anyway. Indexing means that thousands of people will find the secret copy through searches. 48105 WARM SPRINGS BLVD. FREMONT, CA 94539-7498 USA [email protected] 3 Contents Executive Summary ........................................................................... 6 Introduction ...................................................................................... 9 Tips and Tricks Summary ................................................................ 11 Planning Your Recruiting Needs ...................................................... 20 Developing Recruiting Criteria ........................................................... 21 Learning About the Users ................................................................................... 21 Deciding How Many Participants to Recruit ........................................................ 27 Drafting the Screening Criteria ........................................................................... 29 Determining the Appropriate Incentives .............................................. 31 Monetary Incentives .......................................................................................... 33 Non-monetary Incentives ................................................................................... 38 How Incentives May Affect Participant Behavior ................................................ 43 How and When to Provide Incentives ................................................................. 44 Considering the Study Locale ............................................................ 46 When Participants Come to You ......................................................................... 47 When You Must Go to the Participants ............................................................... 50 Planning for Training and Orientation .................................................. 53 Preparing the Screening Script and Questionnaire ................................ 55 Cover Page and Opening Script .......................................................................... 56 Screening Questions .......................................................................................... 58 Invitation and Schedule ..................................................................................... 64 Sample Participant Screening Script and Questionnaire ..................................... 65 Screening and Scheduling Participants ............................................ 72 Working With an Outside Recruiting Agency ........................................ 73 When to Outsource Recruiting ........................................................................... 73 Finding a Recruiting Agency ............................................................................... 74 What to Expect from a Recruiting Agency .......................................................... 76 Choosing a Recruiting Agency ............................................................................ 79 Managing a Recruiting Agency to Your Best Benefit ........................................... 82 Doing Your Own Recruiting ................................................................ 86 Internal vs. External Studies .............................................................................. 86 Finding Participants for Internal Studies ............................................................ 90 Finding Participants for External Studies ........................................................... 91 Making the Calls and Tracking Recruitment Progress ......................................... 96 Reusing Participants ....................................................................... 104 When It Is Okay to Reuse Participants ............................................................. 104 When to Avoid Reusing Participants ................................................................. 104 © NIELSEN NORMAN GROUP WWW.NNGROUP.COM 4 Preparing Participant Session Forms ............................................. 106 Honoring the “Participant Bill of Rights” ....................................... 117 Treating Participants as Human Beings ............................................................ 117 Attending to Participants’ Physical Comfort ..................................................... 121 Ensuring Participants’ Safety ........................................................................... 123 Ensuring Participants’ Privacy .......................................................................... 124 Dealing with Unqualified Participants .............................................................. 126 Future Planning ............................................................................. 129 Building and Maintaining a Participant Database ................................. 129 Building and Managing a Recruiting Staff .......................................... 130 Appendix A: Participant Recruiting Survey .................................... 134 About the Survey ........................................................................... 134 Survey Results .............................................................................. 136 Geographical Areas from Which Participants are Recruited.............................. 136 Who Recruits Participants ................................................................................ 143 Internal Time Spent Recruiting Participants .................................................... 148 Agency Lead Time Required and Fees Charged ................................................ 148 Participant User Groups, Incentives, and Session Length ................................ 150 No-Show Rates ................................................................................................. 158 Pre-Session Interaction ................................................................................... 159 Participant Session Documents ........................................................................ 161 Participant Follow-Up ....................................................................................... 162 Appendix B: Sample Scripts and Forms ......................................... 164 Background Questionnaire ............................................................................... 164 Confirmation Message ...................................................................................... 166 Dealing with Unqualified Participants .............................................................. 167 Eliciting Sensitive Information – Sample Questions ......................................... 167 Eliciting Web Experience - Sample Questions ................................................... 168 Interview Topics for Meeting with the System Team ........................................ 169 Interview Questions for the System Team ....................................................... 170 Minor Consent Form ......................................................................................... 172 Nondisclosure Agreement ................................................................................ 174 Participation Consent Form .............................................................................
Load more

Recommended publications
  • Sleep in Children with Attention-Deficit Hyperactivity Disorder (ADHD): a Review of Naturalistic and Stimulant Intervention Studies
    Pediatric OSA Research Abstracts compiled by David Shirazi DDS MS MA LAc Sleep Medicine Reviews Volume 8, Issue 5, October 2004, Pages 379-402 Clinical Review Sleep in children with attention-deficit hyperactivity disorder (ADHD): a review of naturalistic and stimulant intervention studies Mairav Cohen-Ziona, b, c, 1, , Sonia Ancoli-Israelb, c, San Diego State University/University of California, San Diego Joint Doctoral Program in a Clinical Psychology, San Diego, CA, USA b Department of Psychiatry, University of California, San Diego, CA, USA Veterans Affairs San Diego Healthcare System (VASDHS), Department of Psychiatry, c University of California, 116A, 3350 La Jolla Village Drive, San Diego, CA 92161, USA Abstract Attention-Deficit Hyperactivity Disorder (ADHD) is the most common behavioral disorder of childhood. Multiple clinical and research reports suggest extensive sleep disturbances in children with ADHD, however, current data is contradictory. This paper reviewed 47 research studies (13 stimulant intervention and 34 naturalistic) on ADHD that were published since 1980. The main objectives of this review were to provide pediatric clinicians and researchers a clear and concise summary of published sleep data in children with ADHD, to provide a more accurate description of the current knowledge of the relationship between sleep and ADHD, and to provide current information on the effect of stimulant medication on sleep. Twenty-five of the reviewed studies used subjective reports of sleep, six were actigraphic studies, and 16 were overnight polysomnographic sleep studies (two of which also included Multiple Sleep Latency Tests). All participants were between the age of 3 and 19, and 60% were male.
    [Show full text]
  • Human Performance Regression Testing
    Human Performance Regression Testing Amanda Swearngin, Myra B. Cohen Bonnie E. John, Rachel K. E. Bellamy Dept. of Computer Science & Eng. IBM T. J. Watson Research Center University of Nebraska-Lincoln, USA P.O. Box 704 Lincoln, NE 68588-0115 Yorktown Heights, NY 10598, USA faswearn,[email protected] fbejohn,[email protected] Abstract—As software systems evolve, new interface features common tasks, necessitating longer mouse movements and, in such as keyboard shortcuts and toolbars are introduced. While fact, decrease efficiency. In addition, many toolbars with small it is common to regression test the new features for functional icons may add screen clutter and may decrease a new user’s correctness, there has been less focus on systematic regression testing for usability, due to the effort and time involved in human ability to discover how to accomplish a task over a simpler studies. Cognitive modeling tools such as CogTool provide some UI design. help by computing predictions of user performance, but they still Usability testing has traditionally been empirical, bringing require manual effort to describe the user interface and tasks, end-users in to a testing facility, asking them to perform tasks limiting regression testing efforts. In recent work, we developed on the system (or prototype), and measuring such things as the CogTool-Helper to reduce the effort required to generate human performance models of existing systems. We build on this work time taken to perform the task, the percentage of end-users by providing task specific test case generation and present our who can complete the task in a fixed amount of time, and vision for human performance regression testing (HPRT) that the number and type of errors made by the end-users.
    [Show full text]
  • I Vincitori I Campionati Europei Indoor
    0685-0862_CAP08a_Manifestazioni Internazionali_1 - 2009 11/07/16 11:41 Pagina 824 ANNUARIO 2016 I campionati Europei indoor Le sedi GIOCHI EUROPEI 6. 1975 Katowice (pol) 16. 1985 Atene (gre) 26. 2000 Gand (bel) 1. 1966 Dortmund (frg) 8/9 marzo, Rondo, 160m 2/3 marzo, 25/27 febbraio, 27 marzo, Westfallenhalle, 160m 7. 1976 Monaco B. (frg) Peace and Friendship Stadium, 200m Flanders Indoor Hall, 200m 2. 1967 Praga (tch) 21/22 febbraio, Olympiahalle, 179m 17. 1986 Madrid (spa) 27. 2002 Vienna (aut) 11/12 marzo, Sportovní Hala Pkojf, 160m 8. 1977 San Sebastian (spa) 22/23 febbraio, Palacio de los Deportes, 164m 1/3 marzo, Ferry-Dusika-Halle, 200m 3. 1968 Madrid (spa) 12/13 marzo, Anoeta, 200m 18. 1987 Liévin (fra) 28. 2005 Madrid (spa) 9/10 marzo, 9. 1978 Milano (ita) 21/22 febbraio, Palais des Sports, 200m 4/6 marzo, Palacio de los Deportes, 200m 19. 1988 (ung) Palacio de los Deportes, 182m 11/12 marzo, Palazzo dello Sport, 200m Budapest 29. 2007 Birmingham (gbr) 5/6 marzo, Sportscárnok, 200m 4. 1969 Belgrado (jug) 10. 1979 Vienna (aut) 2/4 marzo, National Indoor Arena, 200m 20. 1989 (ola) 8/9 marzo, Veletrzna hala, 195m 24/25 febbraio, Den Haag 30. 2009 (ita) 17/18 febbraio, Houtrust, 200m Torino Ferry-Dusika-Halle, 200m 6/8 marzo, Oval, 200 m 21. 1990 Glasgow (gbr) CAMPIONATI EUROPEI 11. 1980 Sindelfingen (frg) 3/4 marzo, Kelvin Hall, 200m 31. 2011 Parigi-Bercy (fra) 1. 1970 (aut) 1/2 marzo, Glaspalast, 200m Vienna 22. 1992 Genova (ita) 4/6 marzo, 12.
    [Show full text]
  • WESTFALENREKORDE Stand
    WESTFALENREKORDE und -BESTLEISTUNGEN Stand: 06.08.2017 Verbesserungen im Jahr 2017 sind durch grüne Schrift hervorgehoben. Erläuterungen am Ende der Rekordliste. MÄNNER 100 m 10,01 Julian Reus (TV Wattenscheid 01) 29.7.16 Mannheim 200 m 20,29 Julian Reus (TV Wattenscheid 01) 9.7.17 Erfurt 400 m 44,66 Ingo Schultz (LG Olympia Dortmund) 5.8.01 Edmonton/CAN 800 m 1:43,65 Willi Wülbeck (TV Wattenscheid 01) 9.8.83 Helsinki/FIN 1000 m 2:14,53 Willi Wülbeck (TV Wattenscheid 01) 1.7.80 Oslo/NOR 1500 m 3:33,60 Rüdiger Stenzel (TV Wattenscheid 01) 24.8.97 Köln 3000 m 7:45,1 Harald Norpoth (SC Preußen Münster) 6.6.67 Münster 5000 m 13:14,85 Jan Fitschen (TV Wattenscheid 01) 28.7.07 Heusden/BEL 10 000 m 28:02,55 Jan Fitschen (TV Wattenscheid 01) 4.5.08 Palo Alto/USA 10 km Straße 28:49 Amanal Petros (TSVE 1890 Bielefeld) 6.9.15 Bad Liebenzell Halbmarathon 1:01:18 Michael Fietz (LG Ratio Münster) 4.10.97 Kosice/SVK Mannschaft 3:14:58 TV Wattenscheid 01 2.9.07 Bad Liebenzell Koch 1:03:35, Lubina 1:04:20, Meyer 1:07:03 Marathon 2:10:59 Michael Fietz (LG Ratio Münster) 26.10.97 Frankfurt/Main Mannschaft 7:18:08 LAV co op Dortmund 15.5.83 Frankfurt/Main Spahn 2:17:44, W.Kaderhandt 2:29:59, Kappe 2:30:25 100 km 6:56:25 Werner Endrowait (TuS Iserlohn) 19.4.86 Rodenbach Mannschaft 23:34:19 SuS Schalke 96 22.4.00 Rodenbach Karlsohn 7:12:26, Thamm 7:33:35, Koch 8:48:18 110 m Hürden 13,33 Mike Fenner (TV Wattenscheid 01) 23.6.02 Annecy/FRA 400 m Hürden 48,48 Olaf Hense (LG Olympia Dortmund) 26.6.93 Rom/ITA 3000 m Hindernis 8:15,33 Steffen Brand (TV Wattenscheid
    [Show full text]
  • 1. Can You Explain the PDCA Cycle and Where Testing Fits In?
    1. Can you explain the PDCA cycle and where testing fits in? Software testing is an important part of the software development process. In normal software development there are four important steps, also referred to, in short, as the PDCA (Plan, Do, Check, Act) cycle. Let's review the four steps in detail. 1. Plan: Define the goal and the plan for achieving that goal. 2. Do/Execute: Depending on the plan strategy decided during the plan stage we do execution accordingly in this phase. 3. Check: Check/Test to ensure that we are moving according to plan and are getting the desired results. 4. Act: During the check cycle, if any issues are there, then we take appropriate action accordingly and revise our plan again. So developers and other stakeholders of the project do the "planning and building," while testers do the check part of the cycle. Therefore, software testing is done in check part of the PDCA cyle. 2. What is the difference between white box, black box, and gray box testing? Black box testing is a testing strategy based solely on requirements and specifications. Black box testing requires no knowledge of internal paths, structures, or implementation of the software being tested. White box testing is a testing strategy based on internal paths, code structures, and implementation of the software being tested. White box testing generally requires detailed programming skills. There is one more type of testing called gray box testing. In this we look into the "box" being tested just long enough to understand how it has been implemented.
    [Show full text]
  • Orthogonal Array Application for Optimized Software Testing
    WSEAS TRANSACTIONS on COMPUTERS Ljubomir Lazic and Nikos Mastorakis Orthogonal Array application for optimal combination of software defect detection techniques choices LJUBOMIR LAZICa, NIKOS MASTORAKISb aTechnical Faculty, University of Novi Pazar Vuka Karadžića bb, 36300 Novi Pazar, SERBIA [email protected] http://www.np.ac.yu bMilitary Institutions of University Education, Hellenic Naval Academy Terma Hatzikyriakou, 18539, Piraeu, Greece [email protected] Abstract: - In this paper, we consider a problem that arises in black box testing: generating small test suites (i.e., sets of test cases) where the combinations that have to be covered are specified by input-output parameter relationships of a software system. That is, we only consider combinations of input parameters that affect an output parameter, and we do not assume that the input parameters have the same number of values. To solve this problem, we propose interaction testing, particularly an Orthogonal Array Testing Strategy (OATS) as a systematic, statistical way of testing pair-wise interactions. In software testing process (STP), it provides a natural mechanism for testing systems to be deployed on a variety of hardware and software configurations. The combinatorial approach to software testing uses models to generate a minimal number of test inputs so that selected combinations of input values are covered. The most common coverage criteria are two-way or pairwise coverage of value combinations, though for higher confidence three-way or higher coverage may be required. This paper presents some examples of software-system test requirements and corresponding models for applying the combinatorial approach to those test requirements. The method bridges contributions from mathematics, design of experiments, software test, and algorithms for application to usability testing.
    [Show full text]
  • Cost Justifying Usability a Case-Study at Ericsson
    Master thesis Computer science Thesis ID:MCS-2011-01 January 2011 Cost justifying usability a case-study at Ericsson Parisa Yousefi Pegah Yousefi School of Computing Blekinge Institute of Technology SE 371 39 Karlskrona Sweden 1 This thesis is submitted to the School of Computing at Blekinge Institute of Technology in partial fulfilment of the requirements for the degree of Master of Science in Computer Sci- ence. The thesis is equivalent to 20 weeks of full time studies. Contact Information: Author(s): Parisa Yousefi Address: Näktergalsvagen 5, Rödeby, 371 30 E-mail: [email protected] Pegah Yousefi Address: Drottninggatan 36, Karlskrona, 371 37 E-mail: [email protected] External advisor(s): Helen Sjelvgren Ericsson Address: Ölandsgatan 6 Karlskrona, 371 23 Phone: +46 10 7158732 University advisor(s): Dr Kari Rönkkö, Ph.D Jeff Winter Licentiate School of Computing Internet : www.bth.se/com Blekinge Institute of Technology Phone : +46 455 38 50 00 SE – 371 39 Karlskrona Fax : +46 455 38 50 57 Sweden 2 Abstract Usability in Human-computer Interaction and user- centered design is indeed a key factor, in success of any software products. However, software industries, in particular the attitude of revenue management, express a need of a economic and academic case, for justification of usability discipline. In this study we investigate the level of usability and usability issues and the gaps concerning usability activities and the potential users, in a part of charging system products in Ericsson. Also we try identifying the cost-benefit factors, usability brings to this project, in order to attempt 'justifying the cost of usability for this particular product'.
    [Show full text]
  • SSW 567 Software Testing, Quality, and Maintenance
    School of Systems and Enterprises Stevens Institute of Technology Castle Point on Hudson Hoboken, NJ 07030 SSW 567 Software Testing, Quality, and Maintenance PURPOSE: This memorandum provides the Syllabus for SSW 567 TEXTS: • A Practitioner’s Guide to Software Test Design by Copeland - Artech House [PG] • Software Testing: A Craftsman’s Approach – Jorgensen [ST] • Software Measurement and Estimation by Laird – Wiley (optional) [SME] SOFTWARE: • Open Office/Microsoft Office • Possibly the ATM Simulation software system by Russell C. Bjork [ATM] • Freeware tools for Configuration Mgmt (e.g., Subversion), Test Mgmt (e.g., Testopia) and Defect Mgmt Tool (e.g., Bugzilla, Mantis) COURSE DESCRIPTION: This course is an in-depth study of software testing and quality throughout the development process. The successful student will learn to effectively analyze and test software systems using a variety of methods as well as to select the best-fit test case design methodologies to achieve best results. There is extensive hands-on project work in this course. Emphasis is placed on testing as technical investigation of a product undertaken to determine the system quality and on the clear communication of results, both of which are crucial to success as a tester. COURSE PREREQUISITES: Working knowledge of C, C++, or Java (or some other programming language). Extensive programming will not be required. COURSE OUTLINE SSW 567 School of Systems and Enterprises Stevens Institute of Technology Castle Point on Hudson Hoboken, NJ 07030 Readings in MODULE DESCRIPTION PROJECT Texts Definitions of Testing, QA, Install configuration Lecture 1: Getting [PG] ch. 1, 15, 16; faults, failures & their Costs; mgmt tools, write Started [ST] Ch 1 & 2 Fundamental Issues of testing initial program Lecture 2: SCM Concepts, Processes, [ST] Ch 3: This is Test initial program - Configuration and Tools Continuous background seeded bugs Management Integration.
    [Show full text]
  • Flash Usability Report
    Website Tools and Applications with Flash DESIGN GUIDELINES BASED ON USER TESTING OF 46 FLASH TOOLS By Hoa Loranger, Amy Schade, and Jakob Nielsen Additional research by: Rolf Molich Kara Pernice 48105 WARM SPRINGS BLVD. FREMONT, CA 94539-7498 USA WWW.NNGROUP.COM Copyright © Nielsen Norman Group, All Rights Reserved. To get your own copy, download from: http://www.nngroup.com/reports/website-tools-and-applications-flash/ About This Free Report This report is a gift for our loyal audience of UX enthusiasts. Thank you for your support over the years. We hope this information will aid your efforts to improve user experiences for everyone. The research for this report was done in 2013, but the majority of the advice may still be applicable today, because people and principles of good design change much more slowly than computer technology does. We sometimes make older report editions available to our audience at no cost, because they still provide interesting insights. Even though these reports discuss older designs, it’s still worth remembering the lessons from mistakes made in the past. If you don’t remember history, you’ll be doomed to repeat it. We regularly publish new research reports that span a variety of web and UX related topics. These reports include thousands of actionable, illustrated user experience guidelines for creating and improving your web, mobile, and intranet sites. We sell our new reports to fund independent, unbiased usability research; we do not have investors, government funding or research grants that pay for this work. Visit our reports page at https://www.nngroup.com/reports/ to see a complete list of these reports.
    [Show full text]
  • Standard Glossary of Terms Used in Software Testing Version 3.0 Advanced Test Analyst Terms
    Standard Glossary of Terms used in Software Testing Version 3.0 Advanced Test Analyst Terms International Software Testing Qualifications Board Copyright Notice This document may be copied in its entirety, or extracts made, if the source is acknowledged. Copyright © International Software Testing Qualifications Board (hereinafter called ISTQB®). acceptance criteria Ref: IEEE 610 The exit criteria that a component or system must satisfy in order to be accepted by a user, customer, or other authorized entity. acceptance testing Ref: After IEEE 610 See Also: user acceptance testing Formal testing with respect to user needs, requirements, and business processes conducted to determine whether or not a system satisfies the acceptance criteria and to enable the user, customers or other authorized entity to determine whether or not to accept the system. accessibility testing Ref: Gerrard Testing to determine the ease by which users with disabilities can use a component or system. accuracy Ref: ISO 9126 See Also: functionality The capability of the software product to provide the right or agreed results or effects with the needed degree of precision. accuracy testing See Also: accuracy Testing to determine the accuracy of a software product. actor User or any other person or system that interacts with the test object in a specific way. actual result Synonyms: actual outcome The behavior produced/observed when a component or system is tested. adaptability Ref: ISO 9126 See Also: portability The capability of the software product to be adapted for different specified environments without applying actions or means other than those provided for this purpose for the software considered.
    [Show full text]
  • Remote Web Site Usability Testing - Benefits Over Traditional Methods
    REMOTE WEB SITE USABILITY TESTING - BENEFITS OVER TRADITIONAL METHODS JESSICA GARDNER Statistical Division United Nations Economic Commission for Europe (UNECE)1 Geneva, Switzerland [email protected] Abstract Web sites have become a key communication medium, and usability is an essential factor in good web site design. Usability testing is an inexpensive way to gather valuable feedback from representative users, which can help web designers and content creators make their site more usable and relevant to its audiences. This paper examines the benefits of remote online testing over more traditional face-to-face methods. Remote online testing provides access to a larger pool of potential testers, cuts out travel time, and can significantly lower the cost of usability testing. Although the benefit of face-to- face contact is lost, research shows this method is just as effective in identifying usability issues as traditional testing. The UNECE Statistical Division recently conducted tests of its current web site (www.unece.org/stats) as a basis for redesigning the site’s information architecture and establishing a benchmark for future usability studies. Tests were conducted remotely using online conferencing software, allowing testers to be truly representative of our geographically dispersed users and significantly reducing costs. Keywords: remote usability testing, usability testing, UNECE 1. Testing Web Site Usability – Why and How? Good web site design is generally achieved by maximising two factors: usability and visual appeal [Beaird, 2007]. Usability is important for anything with a user interface, whether it be a potato peeler or a computer application. Factors affecting web site usability include information architecture, navigation, jargon and terminology used, design of user-input forms, and page-layout.
    [Show full text]
  • Causal Testing: Understanding Defects’ Root Causes
    Causal Testing: Understanding Defects’ Root Causes Brittany Johnson Yuriy Brun Alexandra Meliou University of Massachusetts Amherst University of Massachusetts Amherst University of Massachusetts Amherst Amherst, MA, USA Amherst, MA, USA Amherst, MA, USA [email protected] [email protected] [email protected] ABSTRACT test input [74] and a set of test-breaking changes [73], they do not Understanding the root cause of a defect is critical to isolating and help explain why the code is faulty [40]. repairing buggy behavior. We present Causal Testing, a new method To address this shortcoming of modern debugging tools, this of root-cause analysis that relies on the theory of counterfactual paper presents Causal Testing, a novel technique for identifying root causality to identify a set of executions that likely hold key causal causes of failing executions based on the theory of counterfactual information necessary to understand and repair buggy behavior. causality. Causal Testing takes a manipulationist approach to causal Using the Defects4J benchmark, we find that Causal Testing could inference [71], modifying and executing tests to observe causal be applied to 71% of real-world defects, and for 77% of those, it can relationships and derive causal claims about the defects’ root causes. help developers identify the root cause of the defect. A controlled Given one or more failing executions, Causal Testing conducts experiment with 37 developers shows that Causal Testing improves causal experiments by modifying the existing tests to produce a participants’ ability to identify the cause of the defect from 80% of small set of executions that differ minimally from the failing ones the time with standard testing tools to 86% of the time with Causal but do not exhibit the faulty behavior.
    [Show full text]