www.it-ebooks.info “Great book! Whittaker delivers ideas that are innovative, smart, and mem- orable. He really knows how to inspire engineers to think differently about testing.” —Patrick Copeland, Director of Test Engineering, Google “James has perfected a fantastic manual testing methodology. The touring concept not only works, but it works so well that we’ve started sharing the tour concepts in the internal testing courses taught to all of our testers. If you want to bring your manual testing processes into the 21st century then read this book.” —Alan Page, Director, Test Excellence, Microsoft “I began working with James at IBM in 1990. Even then, he was inspiring testers and developers to think outside the box. With this book he’s taken his passion for software quality to a whole new level. Read it and watch yourself become a better tester. James is the real deal and this book should be read by every tester and software developer on the planet who cares about software quality or just wants to have more fun doing what they do.” —Kaushal K. Agrawal, Sr. Director of Engineering, Cisco Systems “James Whitaker is a true visionary in the world of testing. uTest and our global community of QA professionals regularly look to James for inspira- tion, interpretation of trends, and overarching testing wisdom. Now he’s finally written it down for everyone else and our industry will be smarter because of it.” —Doron Reuveni, CEO and Co-Founder, uTest “Only someone like James Whittaker would think of combining the idea of tourism with software testing in such a novel way—and only James could pull it off. The tours approach provides a memorable and extremely effec- tive mental model that combines right degree of structure and organization with plenty of room for exploration and creativity. Bugs beware!” —Alberto Savoia, Google “James is one of the best speakers around on software testing and reading his book is much like hearing him speak. If you want to increase your knowledge of testing and make yourself a better tester, this is the book for you.” —Stewart Noakes, Chairman and Co-Founder, TCL Group Ltd. www.it-ebooks.info “I’ve been doing exploratory testing for some time now and James’ tours have given what I do a name, a focus and more importantly some actual guidance. This book is going to make the job of teaching and performing exploratory testing a whole lot easier.” —Rob Lambert, Senior Test Consultant, iMeta Technologies Ltd “I’m pretty pumped up about this work—it’s sane, it’s new, and I, a normal human, can understand and use it without first studying the combined works of various pompous, dead philosophers. I didn’t have to resort to a dictionary once in the chapters I read. I genuinely feel this work is at the forefront of some long-awaited and sorely-needed evolution for our field.” —Linda Wilkinson, QA Manager, NetJets, Inc. www.it-ebooks.info Exploratory Software Testing www.it-ebooks.info This page intentionally left blank www.it-ebooks.info Exploratory Software Testing James A. Whittaker Upper Saddle River, NJ • Boston • Indianapolis • San Francisco New York • Toronto • Montreal • London • Munich • Paris • Madrid Capetown • Sydney • Tokyo • Singapore • Mexico City www.it-ebooks.info Many of the designations used by manufacturers and sellers to distin- Editor-in-Chief guish their products are claimed as trademarks. Where those designa- Karen Gettman tions appear in this book, and the publisher was aware of a trademark Acquisitions Editor claim, the designations have been printed with initial capital letters or Chris Guzikowski in all capitals. Development Editor The author and publisher have taken care in the preparation of this Mark Renfrow book, but make no expressed or implied warranty of any kind and Managing Editor assume no responsibility for errors or omissions. No liability is Kristy Hart assumed for incidental or consequential damages in connection with or arising out of the use of the information or programs contained Senior Project Editor herein. Lori Lyons The publisher offers excellent discounts on this book when ordered in Copy Editor quantity for bulk purchases or special sales, which may include elec- Keith Cline tronic versions and/or custom covers and content particular to your Indexer business, training goals, marketing focus, and branding interests. For Tim Wright more information, please contact: Proofreader U.S. Corporate and Government Sales Apostrophe Editing (800) 382-3419 Services [email protected] Publishing Coordinator For sales outside the United States please contact: Raina Chrobak International Sales Cover Designer [email protected] Alan Clements Visit us on the Web: informit.com/aw Senior Compositor Library of Congress Cataloging-in-Publication Data: Gloria Schurick Whittaker, James A., 1965- Exploratory software testing : tips, tricks, tours, and techniques to guide manual testers / James A. Whittaker. — 1st ed. p. cm. ISBN 978-0-321-63641-6 (pbk. : alk. paper) 1. Computer software— Testing. I. Title. QA76.76.T48W465 2009 005.1’4—dc22 2009023290 Copyright © 2010 Pearson Education, Inc. All rights reserved. Printed in the United States of America. This pub- lication is protected by copyright, and permission must be obtained from the publisher prior to any prohibited reproduction, storage in a retrieval system, or transmission in any form or by any means, elec- tronic, mechanical, photocopying, recording, or likewise. For informa- tion regarding permissions, write to: Pearson Education, Inc Rights and Contracts Department 501 Boylston Street, Suite 900 Boston, MA 02116 Fax (617) 671 3447 ISBN-13: 978-0-321-63641-6 ISBN-10: 0-321-63641-4 Text printed in the United States on recycled paper at Courier in Stoughton, Massachusetts. First printing August 2009 www.it-ebooks.info This book was written in large part while I was an architect at Microsoft. It is dedicated to all the talented testers who crossed my path while I was there. Thanks, you changed the way I thought, worked, and envisioned the discipline of software testing. Keep up the good work! www.it-ebooks.info This page intentionally left blank www.it-ebooks.info A Fault to Guide Software Testing 010010101011011000100100100101010110110001001001001010101 Table of Contents Foreword by Alan Page xv Preface xvii Chapter 1 The Case for Software Quality 1 The Magic of Software 1 The Failure of Software 4 Conclusion 9 Exercises 9 Chapter 2 The Case for Manual Testing 11 The Origin of Software Bugs 11 Preventing and Detecting Bugs 12 Preventing Bugs 12 Detecting Bugs 13 Manual Testing 14 Scripted Manual Testing 15 Exploratory Testing 16 Conclusion 19 Exercises 20 Chapter 3 Exploratory Testing in the Small 21 So You Want to Test Software? 21 Testing Is About Varying Things 23 User Input 23 What You Need to Know About User Input 24 How to Test User Input 25 State 32 What You Need to Know About Software State 32 How to Test Software State 33 Code Paths 35 User Data 36 Environment 36 Conclusion 37 Exercises 38 www.it-ebooks.info x Contents Chapter 4 Exploratory Testing in the Large 39 Exploring Software 39 The Tourist Metaphor 41 “Touring” Tests 43 Tours of the Business District 45 Tours Through the Historical District 51 Tours Through the Entertainment District 52 Tours Through the Tourist District 55 Tours Through the Hotel District 58 Tours Through the Seedy District 60 Putting the Tours to Use 62 Conclusion 63 Exercises 64 Chapter 5 Hybrid Exploratory Testing Techniques 65 Scenarios and Exploration 65 Applying Scenario-Based Exploratory Testing 67 Introducing Variation Through Scenario Operators 68 Inserting Steps 68 Removing Steps 69 Replacing Steps 70 Repeating Steps 70 Data Substitution 70 Environment Substitution 71 Introducing Variation Through Tours 72 The Money Tour 73 The Landmark Tour 73 The Intellectual Tour 73 The Back Alley Tour 73 The Obsessive-Compulsive Tour 73 The All-Nighter Tour 74 The Saboteur 74 The Collector’s Tour 74 The Supermodel Tour 74 The Supporting Actor Tour 74 The Rained-Out Tour 75 The Tour-Crasher Tour 75 Conclusion 75 Exercises 76 www.it-ebooks.info Contents xi Chapter 6 Exploratory Testing in Practice 77 The Touring Test 77 Touring the Dynamics AX Client 78 Useful Tours for Exploration 79 The Collector’s Tour and Bugs as Souvenirs 81 Tour Tips 84 Using Tours to Find Bugs 86 Testing a Test Case Management Solution 86 The Rained-Out Tour 87 The Saboteur 88 The FedEx Tour 89 The TOGOF Tour 90 The Practice of Tours in Windows Mobile Devices 90 My Approach/Philosophy to Testing 91 Interesting Bugs Found Using Tours 92 Example of the Saboteur 94 Example of the Supermodel Tour 94 The Practice of Tours in Windows Media Player 97 Windows Media Player 97 The Garbage Collector’s Tour 97 The Supermodel Tour 100 The Intellectual Tour 100 The Intellectual Tour: Boundary Subtour 102 The Parking Lot Tour and the Practice of Tours in Visual Studio Team System Test Edition 103 Tours in Sprints 103 Parking Lot Tour 105 Test Planning and Managing with Tours 106 Defining the Landscape 106 Planning with Tours 107 Letting the Tours Run 109 Analysis of Tour Results 109 Making the Call: Milestone/Release 110 In Practice 110 Conclusion 111 Exercises 111 Chapter 7 Touring and Testing’s Primary Pain Points 113 The Five Pain Points of Software Testing 113 Aimlessness 114 Define What Needs to Be Tested 115 Determine When to Test 115 Determine How to Test 116 www.it-ebooks.info xii Contents Repetitiveness 116 Know What Testing Has Already Occurred 117 Understand When to Inject Variation 117 Transiency 118 Monotony 119 Memorylessness 120 Conclusion 121 Exercises 122 Chapter 8 The Future of Software Testing 123 Welcome to the Future 123 The Heads-Up Display for Testers 124 “Testipedia” 126 Test Case Reuse 127 Test Atoms and Test Molecules 128 Virtualization of Test Assets 129 Visualization 129 Testing in the Future 132 Post-Release Testing 134 Conclusion 134 Exercises 135 Appendix A Building a Successful Career in Testing 137 How Did You Get into Testing? 137 Back to the Future 138 The Ascent 139 The Summit 140 The Descent 142 Appendix B A Selection of JW’s Professorial “Blog” 143 Teach Me Something 143 Software’s Ten Commandments 143 1.